When we work on same project for more than one year we strongly start feeling need of automating some tasks. You start thinking of introducing automation testing on the project, if it’s not been considered till now by project management.
Some time it seems to be very boring doing same repetitive tasks again and again and this gives rise to a strong feeling to start with the test automation. But believe me never think to automate 100% of your testing work. User interface, documentation, installation, compatibility and recovery are the areas where testing must be done manually whereas performance testing, regression testing, Load/stress testing are the areas where you can have a chance of reaching near to 100 % automation.
Why automated testing is important?
- Manual Testing of all work flows, all fields , all negative scenariosis time and cost consuming
- It is difficult to test for multi lingual sites manually
- Automation does not require too much of Human intervention. You can run automated test unattended (overnight)
- Automation increases speed of test execution
- Automation helps to increase Test Coverage
- Manual Testingcan become boring and hence error prone.
How to Select Automation Tool for Your Project?
Automation testing is the best way to accomplish most of the testing goals and effective use of resources and time. But you should be cautious before choosing the automation tool. Be sure to have skilled staff before deciding to automate your testing work. Otherwise your tool will remain on the shelf giving you no ROI (Return On Interest). Handing over the expensive automation tools to unskilled staff will lead to frustration. Before purchasing the automation tools make sure that tool is a best fit to your requirements. You cannot have the tool that will 100% match with your requirements. So find out the limitations of the tool that is best match with your requirements and then use manual testing techniques to overcome those testing tool limitations. Open source tool is also a good option to start with automation.
Tool selection is one of biggest challenges to be tackled before going for automation.
Identified requirements and testing type, acts as a base criterion for test automation tool selection.
Define your requirement
During the requirement gathering phase of tool selection one has to list out the requirements for the automation tool.
Prepare some question list and ask yourself.
- What problems do automation tool needs to solve?
- What technical capabilities will the tool need, to be compatible with your environment?
A list of all technologies used in the product and the platforms/ browsers on which the product is supported must be created as the technical requirements for the tool.
For a tool to be able to automate testing of the product, the important criteria is the tool’s ability to identify, access and work with all controls used in application and check if the tool is able to identify and handle all the listed controls.
The list of desirable features that team needs in the tool must be created in consultation with the technical team.
Apart from this, a list of all features supported by the tool must also be created so that all the features of the tool are considered during selecting automation tool.
An example of few features that can be listed are the tool’s support for testing Graphical user Interface (GUI), Application Programming Interface (API) and Command Line Interface (CLI), support to extend tool using APIs like Win32 APIs, verification points supported in the tool, support of an efficient Recovery System and support for test report creation.
Check Compatibility with the AUT (Application Under Testing):
Testing tool need to be compatible with
- The operating systems your product supports.
- The development environments used to create your product.
- Third party software with which your product integrates.
- The test management tool used (in order to be able to integrate with it); this will eliminate data redundancy and will give the advantage of managing all test related data at a single location
Documentation: Check Available documentation for tool as good documentation help in solving lot of problems.
Types of testing
Need to check that automation is suited to which types of testing with specific intentions
- Functional Testing – on operations which perform as per the expectations
- Regression Testing – on the behaviour of the system which has not been changed
- Exception or negative Testing – thereby forcing error conditions in the system.
- Stress Testing – To determine the absolute capabilities of application and Operational infrastructure.
- Performance Testing – To provide assurance that the performance of the system will be adequate for both batch runs and online transactions in relation to business projections and requirements.
- Load Testing – To determine the points at which the capacity and performance of the system become degraded to the situation that hardware or software upgrades would be required.
Check what type of testing problems do you want the Test automation tool to address?
Manual testing problems, Time constraints when implementing small changes in the system, shorter regression testing timeframes
Test data setup, Defect tracking, Training and Support, Teams involved, Licensing cost of the tool, Maintenance cost, Tool’s Extensibility, Tool’s Performance & Stability
Selecting the right tool can be a tricky task. Let’s dive deep and check for the following feature of test automation tool which can help us to select the best tool for our requirement.
Support for various applications and platforms:
In general, the best option is to choose a tool that supports all development tools currently used in your organization, or all tools that you plan on using in the near future. Even if you use one compiler, you might use other compiler in future.
Check whether automated test engine provides extended support for the following applications according to the requirement of your project and organization
C#, Visual Basic.Net or any other .Net application for any version of .Net framework. Java applications, Windows Mobile application, Flex, Flash, Silverlight etc.
Support for various Operating System
Choose an automated testing tool that supports as many operating system as possible. Make sure that the tool supports recent operating system versions. The fact that your customers do not run your application under Windows Vista does not mean that they will not use it in the future. If your application works under this operating system your software testing tool should be able to work under it as well.
Support for various Software Testing types
Check whether your testing tool supports the test types that you are going to run. Can it simulate user actions over the tested application’s user interface i.e. Can it perform functional testing? Can it run unit tests built into the tested application? Does it support distributed testing, load testing and have feature to synchronize test that run simultaneously on several network workstation? There are lot of automated software testing tool that supports only one or two testing types for instance, only unit testing or functional testing.
Easy to use – Creating automated tests without programming and automated test scripts
Many automated testing tools only provide scripting capabilities for creating automated tests. Creating automated test scripts require programming experience and may be difficult for inexperienced testers.
The automated testing tool must be easy-to-use and clear even for inexperienced QA personnel. A good automated testing tool may support several ways for creating automated tests. For instance, it may be provide a simple, easy-to use visual automated test editor and it must let you create tests fast and effectively.
Recording Automated Software Tests
Make sure to select the tool that can record and playback user actions on the tested application. This feature will significantly streamline test development.
Automated test scripts are useful even if the automated testing tool provides easier ways to create automated tests. Scripts are very important when performing automated testing and they let users to create automated tests for specific application features, provide a variety of automated testing actions and perform those operations that cannot be done with standard features built into the automated testing tool.
Automated test scripts are useful even if the automated testing tool provides easier ways to create automated tests, because script code typically provides more powerful functionality.
Support for Controls and Objects
When selecting an automated testing tool make sure it includes special features that simplify the work with controls used in your application.
Supports various Test Data Sources
It is common practice to simulate user input into the tested applications data forms while performing testing. When choosing an automated testing tool, check which data formats it can use such as text files, XML files, database tables and others.
There are some situations where network resources may be unavailable or the desired window is not created, the web page loads very slowly or an unexpected modal dialog box appears. These and other similar factors may cause the automated software testing tool to decide that the test has failed and it should be stopped.
Logging Automated Test Results
Testing automation tool should be able to keep a log of all testing actions and provide a detailed report for them. The log of a software testing tool must contain different types of messages, errors, warnings and informative message. This helps to determine what went wrong and why the test failed.
Extensive Test Reports and exporting reports
Check if the testing tool can export the test log so that the results can be send to your colleagues or team mates, thus automated testing tool should give users the opportunity to export test results to external files.
While selecting a tool take into account the support fees, upgrade price as well as training cost along with the product price. This information can be achieved from vendor only after talking to a representative from their sales department over the phone.
Last but not the least free trial version – don’t forget to use trial version before purchasing the test automation tool.
Choosing the right tools for the job and targeting right areas of the organization to deploy them can only realize the benefits of automated testing tools
Identify Tools – Next task would be identifying different industry-standard tools based on the requirements. This can also include tools to be purchased or open source tools. Identified tools must be evaluated against the checklist.
Checklist – An exhaustive evaluation checklist needs to be created which is in-line with the requirements and the tool is evaluated against this checklist for positive results.
Shortlist the available tools: Good shortlist is between (5-6) tools and then analyse
Create scorecard, Analyse scorecard, try out the best (2-3) tools
Present the selection of tool to the team and get additional input from team.
Sample Run –Many vendors claim that their tool supports specific requirements, but finally when we try creating our scripts it fails. Best way to evaluate is to create a sample run, this includes different types of actions and requirements we need to perform. Create sample scripts and execute the same for results.
Rate and Select Tools – Based on the sample run, supportive tools could be identified and rated. Also there may be scenarios, where multiple tools satisfy the requirements. In the said scenario, we may need to choose more than one tool, for test automation.
Implementation and Training – High rated tools will be procured/open-source licensed. Training needs to be conducted for the project team, on how to use these tools.