How Automated Testing Enables DevOps Efficiency

In the era of fast-changing cloud software landscape, most businesses want to release new software frequently. On one hand, they want to stay up to date or get ahead of the competition and; on the other hand, they want to be able to react to changes and problems rapidly.

Such speed is possible only if their software release process is largely automated. The process must not stop at slow and laborious manual builds or test stages.

DevOps Is a Mindset and a Maturity Journey Rather Than Just a Methodology

DevOps began gaining popularity among the early cloud businesses more than ten years ago and is now present everywhere software is being built and operated. In DevOps, Plan-Code-Build-Test-Release-Deploy-Operate-Monitor repeats in a never-ending and ever-accelerating cycle.

The core idea of DevOps is to deliver fast and frequently, in small increments so that the software keeps evolving rapidly but rarely goes through a massive change at once. This implies eliminating manual work, waiting and handovers among teams. But it also implies a focus on quality. Every bug that passes through the pipeline means rework — and rework means delay.

Today, DevOps is a mainstream delivery model in all types of software teams, from teams building embedded software for IoT devices to teams automating business processes in Salesforce. However, the maturity and capabilities vary by organization. Much like lean or agile, DevOps is a mindset and a maturity journey rather than just a methodology or a toolset.

Most organizations start their DevOps journey by adopting agile software development principles, consistent version control and automated software build pipeline. Those three define what is often called Continuous Integration and Continuous Deployment (CI/CD) although many organizations don’t really deploy automatically.

When You Try to Deliver Fast, You’ll Probably Deliver Defects Too

An optimally working DevOps pipeline is much like a smooth operating manufacturing process: lead times are short, the production line is never idle and the goods flow steadily without piling up at any machine and accumulating work in progress. If problems ever occur, they are easily detected early and corrected before their impact propagates widens in the process.

But optimal pipelines are rare, particularly at the beginning of the organization’s DevOps journey. Companies struggle with the age-old dilemma of whether to choose speed over quality, or vice versa. When you try to deliver fast, you’ll probably deliver defects, too. A bug in production would mean downtime, poor customer experience, financial losses or, in the worst case, even more serious damage.

Bugs in production hurt the DevOps process. Passing too many defects through the pipeline and causing rework will quickly cause the flow to decay. That’s why thorough testing is needed. But more testing means more work that creates a bottleneck in the flow, causing delays and piling up work in progress. Moreover, defects still tend to be detected late in the flow. The solution, obviously again, is automated testing.

Automated testing is a process. It does not come for free — not in the beginning at least. It has to be a part of the automated pipeline. Somebody must actually automate the tests. When the software changes, the automated tests need to be updated accordingly. Chances are that defects are still detected late, although with a much lesser testing effort.

The effort to maintain the automated tests is probably much smaller than what’s required for manual test execution but test maintenance still requires time — and it’s very likely to be needed exactly when the tests are supposed to be run. Automated testing, in its basic form, is thus a way to reduce the cost and execution time of testing but may still cause delays in the overall flow.

Define What Quality Means to Your Customer

The good news is, many businesses have successfully integrated effective testing into their DevOps pipeline, achieving improvements both in speed and quality. The bad news is, there are no shortcuts. One can accelerate a lot by choosing the right tools, but it is even more important to design the whole testing process and upskill the team.

It all starts by defining what quality means to your customers and the minimum tolerable level. Then you can figure out what you need to do to ensure the quality and, as the case may be, improve it. A risk-based approach usually works best. Try to imagine what can go wrong and what the negative consequences would be.

Testing is the most visible, although not the only, part of your quality strategy. Because you want to go fast, you need to automate tests. Still today, human testing tends to be the most effective approach for testing new functions while automated testing is best for ensuring that the old functions still work. But forget about detailed test plans that humans execute; you can automate tests at the same effort. Learn the methods of exploratory testing and apply them to new features, then automate the best parts of your exploration.

Keep Shifting Left

Those who wait to test until the end of the development lifecycle are setting themselves up for unnecessary headaches. The earlier a defect is detected, the less damage and rework it will cause. Encourage your developers to unit test their code before releasing it for integration. With automated tests, you can run them as early and as frequently as possible. With a steady process of creating automated tests and running them in the beginning, you’ll be able to celebrate defects found early.

Make Quality Transparent

If you go to a manufacturing plant, educated workers tend to know their line's production yield and other KPIs. If you go to a software house, the masters of science usually don’t even know how many lines of code they have. Quality will improve when you make it visible, in hard numbers:

• how many defects have been detected in production
• how many before production
• how many known defects remain to be fixed
• how many automated tests there are, and so on

Luckily, testing is easy to measure.

Use Tools that enable collaboration

Modern software teams have technical members, business members, UX members, and many other folks. The more effectively they collaborate, and the more effectively different teams can work together, the better your productivity gets. A multitude of tools, methods, and handovers is poison to speed and quality. Aim at standardizing a few testing tools that serve the purpose and can be used both by technical and non-technical team members.

DevOps Is About Time to Value and Value Is About Functionality and Quality

The trick is to deliver frequently, in small increments. You don’t get much more speed by moving faster — you gain speed by standing less and not having to go back to fix what you left behind. Such an operating mode is only enabled by automation.

In DevOps, software is never ready but it’s always available for use. Testing is the main bottleneck of the DevOp process. Anything you can do to test more and faster will improve both speed and quality, resulting in a shorter time to value.