Fitting Continuous Testing Into Your DevOps Pipeline
Table of Contents
Fitting continuous testing into your DevOps pipeline
As release cycles get shorter and shorter, and development organizations race to keep up and deliver the best and most innovative new software products to their impatient audience, DevOps practices, tools, and processes have become an essential part of the software development lifecycle. Among the numerous DevOps trends that have become a must on organizations’ DevOps adoption list in order to win at the continuous delivery game is continuous testing.
Continuous testing is a way to boost, speed-up, and support the DevOps CI/CD pipelines. It includes the practices, processes, and tools of testing early, testing often, testing everywhere, and automation. According to Gartner, the aim of continuous testing is to “obtain immediate feedback on the business risks associated with a software release candidate.” According to their report, continuous testing means that “testing is no longer a stage in delivery, but an integral DevOps activity that exists in various forms through all areas.”
This is quite a change from the traditional approach to testing and QA. Many companies are still struggling to adopt continuous testing and break away from the olden days when testing was performed at the end of the development cycle. Continuous testing boosts the continuous delivery pipeline by testing at different stages of the software development lifecycle, covering development, integration, pre-release, production, delivery, and deployment.
Continuous testing helps to ensure that quality is baked into the DevOps pipeline. As Jez Humble puts it, the goal “is to run many different types of tests—both manual and automated—continually throughout the delivery process.”
Don’t get it twisted: Test automation vs. continuous testing
As is often the case with new and popular DevOps methodologies, a lot of buzzwords are being thrown around in the continuous testing ecosystem. One of these buzzwords is test automation, another common practice implemented in the continuous delivery pipeline to help support DevOps efforts. Unfortunately, the terms test automation and continuous testing are often confused, or even thought to be interchangeable.
Wikipedia describes continuous testing as “the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” However, many experts take issue with this definition. Lisa Crispin says that while automated tests are definitely an important part of continuous testing, the terms are not interchangeable, and there is much more to continuous testing than simply test automation.
Sushil Kumar, Head of Broadcom’s Continuous Testing Business, agrees, and believes that the Wiki definition is way off, and “misses the essence, nature, and magnitude of the transformation that continuous testing signifies.” Kumar goes on to explain that “while automation is certainly a big part, continuous testing is fundamentally about transforming testing from being a time-boxed event in a linear process to embedding it as a fundamental and ongoing aspect of every activity though the software delivery cycle.”
While test automation is an important part of continuous testing, it is not the only component of this methodology. Continuous testing, like the entire DevOps approach, relies on processes, teams, and tools, and not just on automation.
CT in the CI/CD: Continuous testing throughout the DevOps pipeline
Now that we’ve established what continuous testing is, what it is not, and how test automation fits into continuous testing methodologies, let’s look at how continuous testing fits in throughout the DevOps pipeline.
Sauce Labs, providers of a popular cloud-hosted, web and mobile application automated testing platform, list and explain the functional tests that their most successful customers typically execute throughout all phases of the DevOps pipeline:
#1 Unit testing aims at getting fast feedback on new code. Well-planned and executed unit testing helps ensure that the code is well-structured, with isolated components that are easier to debug and fix later on.
#2 Integration Testing happens when the software components are merged into the shared pipeline, in order to ensure that the build remains stable as new code is introduced.
#3 Automated End-to-End and Regression Testing are performed when the code is finalized, by packaging and deploying it across different servers and resources to ensure that the application works in an environment that imitates production.
#4 Production Testing shifts testing right, extending it after the application is released. Testing in production serves to locate issues before they cause major problems for customers.
#5 Exploratory and Live Testing is when automated testing is replaced by manual, or live, interactive testing, that attempts to break an application or find unexpected results.
Continuous testing best practices
Forrester’s and Tricentis’ Thought Leadership Paper maps out five best practices for implementing continuous testing, based on a survey of 603 global enterprise respondents responsible for their firms’ Agile and/or DevOps strategies.
Forrester research found that companies that have displayed Agile and DevOps maturity invested in these key testing practices when working with their teams, processes and tools to transform their testing into continuous testing.
#1 The organizations allocate proper testing budgets and focus on building and upgrading testers’ skills to implement Agile and DevOps practices across the organization.
#2 They implement continuous testing in response to the demand for faster releases.
#3 They include testers as part of the integrated delivery team.
#4 They automate end-to-end functional testing.
#5 They shift left test and implement it earlier in the development life cycle.
The benefits of continuous testing
Abhinav Kaiser of Pluralsight explains that rather than having testing stuck in a bottleneck and slowing down the SDLC, continuous testing ensures that once the code is integrated it gets tested automatically. In this way, continuous testing supports the DevOps goal of speeding up the software delivery process to deliver high-quality software.
Continuous testing also helps save a lot of time and efforts that developers used to invest in waiting for the QA teams’ to complete testing, and then having to go back and fix their code. Kaiser explains that continuous testing cuts down the feedback cycle significantly and helps avoid the time consuming and tedious task of rewriting pieces of code.
In addition to achieving faster release cycles, continuous testing, when done right, helps to ensure improved quality, because everything is tested on the build, or even earlier.
Another benefit that comes from continuous testing pipelines is minimizing risk. Repeatedly checking different aspects of the software projects from the earliest stages of development until delivery promises testers, developers, and DevOps teams that they are minimizing risk every step of the way.
Ensure continuous testing success
The 2019 Report on continuous testing, showed that 58% of enterprises deploy a new build every day, with 26% even deploying at least hourly. However, while nearly a third of the 500 IT executives that participated in the survey say their departments had “fully embraced continuous testing”, there are still quite a few obstacles to continuous testing adoption and success.
Unfortunately, one of the main challenges that organizations still face is automation, as the study showed that automation has yet to be fully embraced throughout the DevOps pipeline. In order to succeed at continuous testing, organizations must embrace all aspects of the DevOps approach. That means investing in all of the teams involved, including DevOps, developers, and in this case the testing teams. In addition, tools and automation need to be adopted in all phases of the CI/CD pipeline. Finally, new processes need to be implemented and adopted by everyone in the development organization, in order to ensure the success of a well-orchestrated continuous testing process.
Knowing is the first step to facing the many challenges involved in the complex process of successful implementation of continuous testing. While there are still many organizations that are in the process of adopting continuous testing and working through the bumps in the road, wise investment in the tools, the people and the processes that bring it all together can help overcome the barriers and achieve continuous testing success.
