We will now discuss one of the most misunderstood subjects inside most organizations today and that is the subject of automated testing. There are so many misconceptions regarding automated testing, especially regarding the types of automated testing to do and what to use. Below we will discuss many of the myths and misinformation regarding automated testing.
The first bit of misinformation we will discuss is the individuals and vendors doing tool comparisons of functional automation tools. If you come across anyone doing a tool comparison between say a commercial functional automation tool and either Selenium or Protractor, our best advice is to close the site and disregard anything you may have read. You are probably asking why we suggest this advice, the reason is that they are comparing apples and oranges. The commercial functional automation tools are just that, a set of tools for functional automation testing. Selenium and Protractor are API’s and nothing more, the commercial tools often come with their own IDE, test repository, ability to run the tests and a limited number of languages to write the tests in. On the other hand, an API like Selenium can be written in almost any programming language and uses the languages corresponding xUnit framework tool to run the tests and report the results. You may be asking yourself, well wouldn’t we want all these tools the commercial vendor offers, the answer is probably not. The reason is fairly straight forward, if you are for example a Java shop, wouldn’t you want to be able to use the same IDE you write your application code to also write the automated tests, and wouldn’t you want to be able to write the tests in Java if that is what most of your resources are skilled in? There are a fairly large set of other reasons you probably would not want to go with the commercial vendors if you are moving to DevOps, we will highlight a few more below.
Another mistaken misconception comes from users of many of these commercial tools. When discussing an API like Selenium and the use of Page Objects you will often hear them state that their tool is superior because it can generate their version of Page Objects for them and Selenium users have to manually create their Page Objects. The problem is that these commercial users are completely blind to the move to start also start writing even functional automated tests in a Test-Driven Development manner. Using Page Objects with Selenium and especially when combined with Specification by Example allows the test writers to basically stub out the Page Object and tests in the same manner as a Unit test and then code to make them run. Many of the commercial tools require the code to be written before they can generate their version of Page Objects, making any attempt at Test Driven Development futile.
A couple of other myths are that organizations can employee a record and play tool for automated functional testing and achieve DevOps and that traditional Quality Assurance can take a quick two or three day class and become full blown automated testing engineers. The first myth regarding record and play, you will never in any situation be able to accomplish DevOps with a record and play tool. The maintenance cost alone will make this completely impractical. The other myth that traditional Quality Assurance can take a single class and suddenly become automation engineers is simply not reality for the majority of Quality Assurance. Make no mistake, in order to do automated testing, one must possess programming level skills to do it properly.
The last misconception made by many organizations who are moving towards DevOps is that they simply need to incorporate Unit testing and some functional automated testing and they are ready for DevOps. Organizations who do DevOps actually employee automated Unit testing and possibly mutation testing on top of that to test the Unit tests, functional automated testing, automated security testing, backend and frontend performance testing, web consistency testing, automated accessibility testing and automated infrastructure testing. Most of the automated testing required can be found in traditional Agile/DevOps testing pyramid.