Authored by Thom Behrens, DevOps Expert
Renowned systems thinker W. Edwards Deming said of quality control: “Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.” From his influence, the concept of “Built-in Quality” has become a key tenet of Lean software development and DevOps. Testing through inspection at the end of a software product’s life cycle has proven to be inefficient: bugs discovered at this stage require developers to dredge up initial requirements and revisit long-forgotten work, changes can have cascading effects on dependent modules, and this often leads to expensive delays to the overall project.
Within software engineering, we can shift from checking for quality at the end of a process to implementing Built-in Quality by integrating continuous feedback throughout the development process, automating testing, and encouraging “ownership thinking” for all members of the team. Much has already been written about overcoming the disadvantages listed above. And there are many more advantages of adopting continuous feedback: shorter lead times, more stability at a module level, less re-work and wasted work, and improved morale.
But even while many organizations have made progress towards integrating continuous feedback into their deployment streams, they may still be delaying security & compliance testing until the end. DevOps principles which take into account the importance of security and compliance – referred to as DevSecOps – can help us approach security & compliance in the same way we see functional quality, and strive for “Built-in Security” instead of “Security Checked For”, as is common practice in many organizations.
Many potential security vulnerabilities can be caught even before code is committed, by inline static analysis tools which are able to help sanitize user input, enforce database access patterns, or prevent cross-site scripting. But like any other bug, security vulnerabilities often manifest as the culmination of multiple implementation decisions that combine to produce unanticipated results. As such – and like all testing – security testing isn’t complete until comprehensive penetration tests have been run against the full stack.
We often confuse things in the development process which require the entire stack with things in the development process which need to happen last. A single-stream, unidirectional delivery strategy – the “Waterfall” method – implies that access to the full stack isn’t available until the project is “dev complete”, but not so with Continuous Delivery. By acknowledging that software products never truly reach a state of “dev complete”, we instead embrace the policy of always having our product in a deployable state, every day starting today. When we achieve this, we can test the entire stack as we deliver incremental changes, and deliver continuous improvement of security and compliance measures as we do so.
It’s important to recognize that just as we won’t accidentally build great features, we won’t accidentally achieve a secure system. Security is something that needs to be considered from the beginning and not just considered at the end of the development process. As Anders Wallgren says, “security is not like some condiment that you sprinkle on top of the plate before you serve the food. Security is a core ingredient in the meal.”
For new projects, keeping the stack in a constantly deployable stack from day 1 starts with shipping some workable feature, and then incrementing from there. Concepts like the Hello Production strategy provide illustrations of how to get this done. For mature software products already in production, many books have already been written how to implement Continuous Delivery, but strategy for handling security vulnerabilities within mature systems need to be understood as an I/O system – it’s just as important to remove existing critical security risks from the system as it is to prevent new critical security risks from being introduced. Prioritizing vulnerabilities and then iteratively implementing fixes & tests for those vulnerabilities is a well-worn pattern. Key retrospective questions are “what did we do to become safer this week?” and “what did we do to become faster this week?”. When done in conversation with all the necessary stakeholders, results along these success metrics will grow together.
When our stated goal becomes to keep our application in a constantly deployable state, or getting things into production as quickly as possible, it is worth mentioning that Cloud-provided IaaS or PaaS options bring benefit compared to traditional app stacks. Infrastructure-as-code built on infrastructure clouds provides an easy way to make the final product more dependable. And PaaS solutions like Salesforce, further reduce the vulnerability and management burden for your solution. Salesforce also brings the benefit of an app ecosystem including rules engines like Copado that let you build industry-standard or organization-specific compliance checks into your process.
Security and regulatory compliance is a critical ingredient for any modern organization, whether you’re aiming for success with customer trust & responsible data stewardship, or looking to avoid hefty fines. But the solution to the problem doesn’t have to be as painful as the consequence. Adopting “Built-in Security” as a mantra and a model to strive for can work alongside and in support of your mission to establish continuous delivery.