Comparison: Test Case Reporting, Management and Metrics
As part of an ongoing blog series, we will delve into a discussion of the differences between an ATF and the various commercial products on the market. This blog series will discuss one area of comparison each week. This week’s discussion will be on Test Case Reporting, Management and Metrics.
The commercial tools have long championed their ability to not only test software, but also have their own integrated quality reporting tool set to report on test cases, manage them and provide metrics. Looking at this fairly there are advantages and disadvantages to having your own proprietary quality/reporting tool sets and test case management, they generally will easily integrate with the testing tool set, and they provide good reporting, metrics and test case management. The disadvantages are that they are expensive, are proprietary and are often not best in class. Recently one of the major commercial providers has announced how they have integrated with Selenium/WebDriver and now you can run your tests from their proprietary test case management tool and get the results in their reporting tool set. The general message from this commercial provider is that now Selenium/WebDriver is a useful tool because you can get the test case results reported.
For some strange reason the commercial tool makers keep making the argument that Selenium does not have any reporting, metrics or test case management built into its tooling and thus is an inferior product. This is actually an advantage because the Selenium project concentrates on what it does best, provide a best in class web based tool set. Again, the commercial tool makers try to make a comparison with the record and play Selenium IDE instead of the code based Selenium/WebDriver test tool, which actually does have test case reporting provided by the xUnit framework that drives the ATF. As an example if your ATF is built around Java then you may be using TestNG which not only provides a very good test case reporting format, but the reporting from TestNG can be automatically uploaded to any web site and customized or emailed. If you happen to be a .Net shop then you have TFS which can also monitor an automated test run and report on the results.
The advantage on the side of Selenium/WebDriver is that it is open source and can easily integrate with other best in class open source tools. An example of this is the Test Case Management tool TestLink, you can easily integrate your automated test cases in your ATF with TestLink and get test case tracking, reporting, pass fail analysis and historical test trend analysis after each automated test run. The biggest advantage an ATF has that none of the commercial tools have is the ability to integrate with a tool such as SonarQube to manage the code quality of your ATF.
In the comparison of Test Case Reporting, Management and Metrics the ATF side is at worst even with, if not slightly ahead due to SonarQube when compared with the commercial tools on the market.