Automated regression testing: Effective strategy

Editor’s note: If you’re unsatisfied with the quality of your software or experience troubles with sticking to release deadlines, you might have taken a wrong approach to regression testing. Continue reading the article and don’t hesitate to turn to our software testing services to streamline your regression testing with test automation.

Regression testing assures that code changes introduced to software haven’t altered or broken the already existing functionality. I find this type of testing of particular importance for projects following Agile, where code changes are frequent and ongoing.

However, regression testing is monotonous and time-consuming. To optimize the regression testing process, I often advise leveraging test automation. In one of our projects, for instance, testing performance achieved with automated regression testing equaled the performance of 15 manual test engineers.

Still, applying test automation doesn’t necessarily make all the pains of regression testing go away. An erratic automate-it-all approach may only increase testing efforts without bringing much gain. To implement regression testing automation efficiently, I recommend designing a comprehensive test automation strategy tailored to the application and project specifics. I’ve summed up the cornerstones of the test automation strategy we follow at ScienceSoft below.

4 pillars of ScienceSoft’s regression testing automation strategy

Devising a regression testing automation strategy, we outline the answers to the following core questions:

1. Does test automation suit the project?

Test automation, including automation of regression testing, doesn’t suit all projects equally well. I find it effective to introduce test automation in large- or at least medium-scale projects involving several sub-systems (e.g., enterprise web applications, multi-user games with numerous releases). For small and short-term projects (e.g., simple mobile apps), automation usually doesn’t make sense as the time required for automation may well exceed the planned project time frame.

2. When to start scripting?

When automating regression testing, it’s impossible to write scripts beforehand. Regression test automation relies on efficient manual test cases that repeatedly found bugs in the past, and these are not available at the start of the project. Thus, at ScienceSoft, we roll out regression testing automation when some software modules are already stable, and then proceed with automating regression testing of other stabilized modules.

3. Which tests to automate first?

We develop regression test scripts based on repeatable manual regression test cases that consistently found bugs. As a rule, these tests cover core product functions and exceptions, such as negative test cases or boundary conditions that may affect adjacent functional areas of the application under test.

4. What percentage of regression testing can be automated?

In the majority of our projects, regression testing scripts cover 70 – 90% of repeatable and effective manual test cases that stably detect bugs. The remaining 10 – 30% are manual test cases that detected bugs only once or reported false positives or negatives.

Recommended frameworks for automating regression testing

For automating regression testing, ScienceSoft prefers Selenium WebDriver 3.0, Appium, and Protractor. Still, the right tool depends on the peculiarities of software under test. To get a better idea of which tool suits your application best, check out this list of useful tools in automated testing prepared by my colleague Alexander Viktarau, test automation architect with two decades of experience.

Having trouble with regression testing?

ScienceSoft’s test automation engineers are ready to develop a structured test automation strategy and execute relevant test automation activities to streamline your regression testing.

Enhance regression testing

How we improve automated regression testing

Depending on the project, ScienceSoft’s test automation team chooses data-driven and/or keyword-driven approaches to testing.

Applying data-driven testing (DDT), we separate test scripts from test data, and store the latter as a table with inputs and expected results. This way, we can run a script for a sizeable set of data from a relevant table. I find it effective to apply the data-driven approach for regression testing of data-intensive applications (e.g., ERP, CRM, BI applications) as it helps address the problem of changing and/or expanding test data. Whatever the data in the class, the script doesn’t change, provided that the application business logic stays the same.

Let’s consider a simple example of a data-driven script for setting up Admin accounts in a system:

1 – the script; 2 – the function that sends test data from the table to the script; 3 – the table of test data

Applying keyword-driven testing (KDT), we compile a list of keywords associated with the specific actions in a test case. The keywords take the form of simple step-by-step instructions, for instance:

As the sequence of keywords drives the script, we can use the same list of keywords to build a variety of test scripts. We apply the keyword-driven approach for the projects catering to a large audience and requiring a broader focus. The reason is that compiling keywords doesn’t require scripting skills, therefore, manual testers, BAs, project managers, industry experts, and key stakeholders can contribute their unique perspective to the testing process.

To improve the quality of automated regression testing in a project, we may go for a suitable combination of the two approaches, if the project scale and time frame allow it.

The pitfalls of automated regression testing we keep in mind

To make regression testing automation robust and effective, our test automation team approaches it keeping in mind the common obstacles that may hamper automation efforts.

Maintenance efforts

An automated regression test suite cannot be valid forever. To be efficient, this test suite should reflect any changes introduced to software with the shortest delays possible. ScienceSoft’s test automation team regularly reviews automated regression test suites to detect obsolete test cases that no longer fit the project.

False positives

False positives are tests that report a failure when an application doesn’t have defects in a validated module. False positives are typically caused by obsolete test suites and timing issues (for example, when the testing tool ‘clicks’ buttons on a page that hasn’t loaded completely). To minimize false positives, I recommend verifying automated regression testing results with a quick manual smoke test.

Don’t let regression testing be a burden!

Regression testing automation helps secure the project quality and reduce efforts for tedious tasks. However, automating regression testing is a complex effort that requires an effective testing strategy and expertise in test automation. You are welcome to leverage our 35 years of experience in software testing and leave the trouble of designing a test automation strategy and performing test automation activities to ScienceSoft. If you are interested, please let me know.