Comparison: Test Case Run Time
As part of an ongoing blog series, we will delve into a discussion of the differences between an ATF and the various commercial products on the market. This blog series will discuss one area of comparison each week. This week’s discussion will be on Test Case Run Time.
Let’s start the comparison of Test Case Run Time with a discussion I had a few months ago with the automation lead of a large Fortune 500 company who was trying to tell me about all of the advantages of the large commercial testing tool they used internally. The automation lead was telling me all of the usual perceived advantages of the commercial tool, but then he said something that caused me to stop him mid-sentence and ask him to repeat what he had just told me. The automation lead told me that the set of test cases that made up their smoke test suite took almost an entire day to run. I was in total disbelief when I heard this and asked him to clarify exactly what these test cases were testing, he explained that they just made sure all of the pages opened and that the basic workflow still worked.
In looking for advantages of Test Case Run Time on the commercial tool side there are absolutely none, and worse is that the long run times encountered make it impossible for an organization employing one of these solutions to truly be agile. The commercial tool vendors can tell you any story they want about their products being agile friendly, but it is simply impossible to truly be agile when you have automated test case runs that can take a day or days to run.
On the ATF side the advantage is obvious, the ATF side is capable of spinning up thousands of test cases running against multiple browsers types and browser/OS versions in a matter of minutes. The ATF side easily integrates with leading edge CI tools and sophisticated agile organizations can spin up full test environments in the cloud on demand along with the leading cross browser testing platforms to perform full test runs in a matter of minutes. Simply looking at Test Case Run Times from and agile point of view its obvious which tool wins and should be employed.