There are a number of automated test tool comparisons available on the web today comparing Selenium against various commercial products and they all tend to show the commercial products in positive light compared to Selenium. If one looks closely at these comparisons they will start to see some falsities in these comparisons and if one looks even closer one sees that they tend be from individuals who either do not possess the knowledge necessary to make a fair comparison or the person doing the comparison has ulterior motives. Below are some of the common false or incorrect comparisons made.
Most comparisons done try to compare Selenium IDE against commercial tools, the problem is that Selenium IDE is the record and play version of the tool and no professional would ever suggest using Selenium IDE. The comparisons should be actually using Selenium/WebDriver, which is not a record and play tool unlike the commercial tools. Record and play tools should never be used due to the cost of maintaining the scripts and their practice of hard coding test data.
Another false comparison made is that Selenium does not have an easy and robust IDE built in to create scripts. Since Selenium/WebDriver is created using the users preferred programming language driver, the user is free to use one of many leading edge IDE’s available. These professional programmer IDE’s are far superior to the built in IDE’s used by the commercial tools.
The comparisons being made always try to down play the fact that Selenium’s programming language support is far superior to commercial tools. The commercial tools only support VB Script, VB.NET and C#, while Selenium supports Java, Ruby, Python, C#, Perl and PHP. Ask yourself this simple question, if you are a Java shop wouldn’t you want to use a test tool that allows you to leverage your Java expertise?
Many of the comparisons also trumpet the commercial tools support for objects and Selenium does not. Again this is a misleading comparison, Selenium uses the concept of Page Objects which is superior to the automatic recording of objects by the commercial tools. Because Selenium/WebDriver is not a record and play tool, but a true scripting tool, automation developers can create much more robust methods on the Page Objects to interact with these objects on web pages. The big secret the commercial products do not want you to know is that through the use of Page Objects and Behavior Driven Development Selenium/WebDriver tests can be created in a TDD manner because of the ability mock and easily refactor the Page Objects and test cases. The commercial products cannot easily be used in a TDD manner because of this same automatic recording of objects they are trumpeting, simply put the tools cannot record objects or create tests until the application code has been written.
One of the more interesting claims made in the comparisons is that Selenium does not support or easily integrate with ALM and CI tools. These comparisons try to shine a positive light on the built in integration with ALM and CI tools the commercial products have, the problem is that they do not tell you that they only integrate with their ALM and CI tools and they cannot work with the industry leading ALM and CI tools which Selenium uses. Yes, Selenium tests can be run by the industry leading CI tools such as Jenkins and Bamboo and the results can be reported to ALM tools such as Rally and CollabNet.
Lastly the comparisons love to point out that unlike the commercial tools, Selenium can only support web applications while the commercial tools support web, windows, mobile and performance. Once again the individuals have left out some critical information. Selenium is just one piece of an agile testing framework, meaning that other tools easily plugin to the framework to handle windows, mobile, services, database and performance testing. The agile testing framework using Selenium is actually superior in the comparison of application support because all the various testing types are built into one framework, while the commercial tools require multiple tools that are not integrated in order to support all of these application types.