Go to ScienceSoft Global
How does Continuous Testing Work in Practice?
Editor’s note: In the article, Tatyana shares proven continuous testing practices we implement for our clients to help them achieve their quality goals. If you decide to implement the practices described or feel like you need a deeper engagement of QA consultants, you are welcome to consider our software testing and QA services.
At ScienceSoft, we realize continuous testing by incorporating incremental, uninterrupted testing activities throughout the delivery pipeline. This way, we validate the quality of small changes made to an application, and detect and prevent defects as early as at the requirements engineering and software design stages. Analyzing the results of applying continuous testing practices in my projects, I can say that introducing even the elements of it helped us increase the test process velocity by 25% on average and improve software quality by 30%.
Below, I summarized the cornerstones of ScienceSoft’s approach to continuous testing and listed the practices we implement when setting up continuous testing for the customers who outsourced QA to ScienceSoft.
ScienceSoft’s approach to continuous testing
I divide all continuous testing activities into those we take before the actual development starts and those we incorporate directly in the development process.
Preparatory activities
Splitting software into modules
Large, tightly coupled applications comprising multiple components that need to be compiled and deployed together pose difficulties to continuous testing. To get fast feedback on the quality of such apps, we at ScienceSoft break them down into modules. We single them out based on the functional similarity (for instance, dividing an e-store into such modules as user account and catalog management, checkout and payment, analytics and reporting, and others) and prioritize them based on business risk to test and deploy independently.
Taking an optimized approach to the infrastructure organization
The incremental nature of continuous testing requires a special approach to the infrastructure organization, namely, setting up a Continuous Integration/Continuous Delivery pipeline. The CI/CD pipeline enables us to roll out unit tests as well as automated API and UI tests throughout the software delivery life cycle and trigger the execution of these tests without human involvement. My colleague Andy Lipnitski, a DevOps consultant at ScienceSoft, says that his tools of preference for setting up Continuous Integration/Continuous Delivery pipelines are Jenkins, Bamboo, and GoCD.
Continuous testing practices incorporated during development
Increasing the share of testing at the unit and API levels
Software can be validated at the unit, API, and UI levels. And in a traditional testing process, a larger share of tests lies on test teams who manually validate software at the UI level. In such case, the software testing pyramid looks like this:
As for continuous testing projects, at ScienceSoft, we shift a larger share of testing activities to the left, involving the development teams in the QA process, which makes the software testing pyramid to look like this:
My colleague Vadim Belsky, Head of Web Development at ScienceSoft, explains the need for increasing the volume of unit testing: “Being relatively easy to write and taking seconds to execute, unit tests provide a reliable insight into the quality of changes introduced to an application and help prevent bugs from entering later stages of the delivery cycle. To make sure that the results of unit testing are reliable, we keep unit tests regularly updated and include them into a common application code review practice.”
Also, in projects with continuous testing characterized by short release cycles and frequent changes, UI tests are quite difficult and expensive to maintain. Thus, we use API tests as a primary means of validating changes made to the application logic. Depending on the specifics of the project and application, I can opt for manual or automated API testing, or choose the combination of both.
Automatically triggering test execution
As I have said earlier, we often integrate test automation frameworks with automation servers like Jenkins in our continuous testing projects. This enables us to trigger automatic test execution and run tests in parallel. For instance, validating a complex image-processing application for our customer, we employed several agents to automatically set out unit and UI tests. Unit tests were executed upon each new code commit, and a partial regression test suit comprised of 470 automated UI tests – within 15 minutes after a new code commit, with the number of commits amounting up to 30 a day per developer.
Thoughtfully combining manual testing and test automation
Despite a common belief, switching to continuous testing doesn’t equal eliminating manual testing. At ScienceSoft, we take an integrated approach to testing and combine manual and automated efforts. I choose to automate testing of the functions that do not change much with each iteration, while the automation of complex or frequently changing functions may take up too much effort and is likely to outweigh the promised benefits. Performing manual testing, we validate critical software modules earlier in the delivery cycle by employing sophisticated testing techniques, for instance, decision tables.
What can you achieve with continuous testing?
Based on ScienceSoft’s experience with continuous testing, employing the techniques described, you are likely to get the following benefits:
- Quick feedback about software quality. Automated testing activities implemented throughout the software delivery life cycle allow evaluating each layer of an application’s architecture and obtaining fast and frequent feedback about the quality of software under test.
- Faster releases and shorter time to market. Continuous testing practices increase testing velocity and help us reduce the time it takes to validate software. For example, in one of our projects, UI-level test automation alone reduced the testing time by at least 25%.
- Communication transparency among project teams. Implementing continuous testing practices, it is possible to eliminate silos between the testing, development, and operations teams. The ongoing communication allows delivering new software pieces to production faster and with no critical defects.
Don’t leave defects a chance!
My practice shows that continuous testing can drive impressive quality and efficiency outcomes. However, the implementation of continuous testing practices is often bound to challenges. First, a project team needs to adapt to the DevOps culture, which requires all team members to work in alignment. Also, the specifics of the application itself can cause difficulties in adopting continuous testing – the practices described may be challenging to implement in projects dealing with software having a complex architecture and business logic.
My colleagues and I have helped many of ScienceSoft’s customers to transition to continuous testing so that now, their test process drives tangible quality and efficiency improvements. In case you consider transitioning to continuous testing, too, you are welcome to leave us a request for assistance.