From Securing DevOps by Julien Vehent This article discusses how important security is to the DevOps process.

Save 37% on Securing DevOps. Just enter code fccvehent into the discount code box at checkout at manning.com.

The DevOps approach

“DevOps is the process on continuously improving software products through rapid release cycles, global automation of integration and delivery pipelines and close collaboration between teams.”

The goal of DevOps is to shorten the time and reduce the cost of transforming an idea into a product that customers use. DevOps makes heavy use of automated processes to speed up development and deployment. Figure 1 shows a comparison of a traditional approach of building software, at the top, compared to DevOps, at the bottom.

• In the top section, the time between idea and availability to customers is eight days. Most of that time is consumed by the infrastructure deployment, as engineers need to create the components needed to host the software on the Internet. Another big time consumer is testing and review between the deployment steps.
• In the bottom section, the time between idea and delivery is reduced to two days. This is achieved by using automated processes to handle the infrastructure deployment and software test/review.

Figure 1: DevOps reduces the time between the concept of a feature and its availability to customers.

An organization able to build software four times faster than its competitor has a significant competitive advantage. History has shown that customers value innovative products that may be incomplete at first, but improve quickly and steadily. Organizations adopt DevOps to reduce the cost and latency of development cycles, and answer their customer’s demands.

In DevOps, developers release new versions of their software, test them, and deploy them to customers in as short as a few hours. Figure 2 zooms in on the bottom section of figure 1 to detail how the techniques of Continuous Integration, Continuous Delivery, and Infrastructure as a Service are used together to achieve fast release cycles.

The key component of the pipeline in figure 2 is the chaining of automated steps to go from a patch being submitted by a developer to a service deployed in a production environment, in a completely automated fashion. Should any of the automated steps fail along the way, the pipeline is stopped and the code is not deployed. This mechanism ensures that tests of all kinds must be passed before a new version of the software can be released in production.

Figure 2: Continuous Integration (CI), Continuous Delivery (CD), and Infrastructure as a Service (IaaS) form an automated pipeline that allows DevOps to speed up the process of testing and deploying software.

Test Driven Security

The myth of attackers breaking through layers of firewalls or decoding encryption with their smartphones makes for great movies, but poor real world examples. In the majority of cases, attackers go for easy targets: web frameworks with security vulnerabilities; out of date systems; administration pages open to the Internet with guessable passwords; or security credentials mistakenly leaked in open source code. Our first goal in implementing a continuous security strategy is to take care of the baseline: apply elementary sets of controls on the application and infrastructure of the organization and test them continuously. For example:

• SSH Root login must be disabled on all systems
• Systems and applications must be patched to the latest available version within 30 days of its release
• Web applications must use HTTPS, never HTTP
• Secrets and credentials must not be stored with application code, but handled separately in a vault accessible only to operators
• Administration interfaces must be protected behind a VPN

The list of security best practices should be established between the security team and the developers and operators to make sure everyone agrees on their value. Rapidly, a list of baseline requirements can be assembled by collecting those best practices and adding some common sense.

Application Security

Modern web applications are commonly exposed to a wide range of attacks. The Open Web Application Security Project (OWASP) publishes a ranking of the top 10 most common attacks every three years[1]: Cross-site scripting; SQL injections; Cross-site request forgery; brute force attacks; etc. The list goes on and on. Thankfully, each attack vector can be covered using the right security controls in the right places.

Infrastructure Security

Relying on an IaaS to run software does not exempt a DevOps team from caring about infrastructure security. All systems have privileged access points, like VPNs (if you have not installed a VPN, there are different posts that discuss cheap VPNs compared with each other to help get you started), SSH gateways, or administration panels. When an organization grows, special care must be taken to continuously protect the systems and networks while opening new accesses and integrating more pieces together. While I was trying to develop a better understanding of growing a businesses cybersecurity, a previous colleague was telling me about he came to understand how valuable a VPN was for his small business he recently started. Before he started using VPNs, he discovered that even at a smaller scale his companies files were valuable to cyber attacks and lost sensitive data through a vulnerability in his firewall. After this breach, he decided to make use of a free vpn trial to measure the benefits of increasing his cybersecurity. From what he tells me, he hasn’t had any issues since. This would be coupled well with a proxy provider to ensure total security.

Pipeline Security

The DevOps way of shipping products through automation is vastly different from traditional operations most security teams are used to. Compromising a CI/CD pipeline can grant an attacker full control over the software that runs in production. Securing the automated steps taken to deliver code to production systems can be done using integrity controls like commit or container signing.

Testing continuously

In each of the three areas defined above, the security controls we implement remain fairly simple to apply in isolation. The difficulty comes from testing and implementing them everywhere and all the time. This is where Test Driven Security comes in. TDS is a similar approach to Test Driven Development (TDD), which recommends that developers write tests that represent the desired behavior first, then write the code that implements the tests. TDS proposes to write security tests first, thus representing the expected state, and then implement the controls that pass the tests.

In a traditional environment, implementing TDS is difficult because tests must run on systems that live for years. But in DevOps, every change to the software or infrastructure goes through the CI/CD pipeline, which is a perfect place to implement TDS, as shown in figure 3.

Figure 3: Test Driven Security integrates in CI/CD to run security tests ahead of deployment in the production infrastructure.

The TDS approach brings several benefits:

• Writing tests forces security engineers to clarify and document expectations. Engineers can build products with the full knowledge of the required controls rather than catching up post-implementation.
• Controls must be small and very specific units, which are easy to test. Vague requirements such as “encrypt network communication” are avoided, instead we will prefer the explicit “enforce HTTPS with ciphers X, Y and Z on all traffic”, which clearly states what is expected.
• Re-usability of the tests across products is high, as most products and services share the same base infrastructure. Once a set of baseline tests is written, the security team can focus on more complex tasks.
• Missing security controls are detected prior to deployment, giving developers and operators an opportunity to fix the issues before putting customers at risk.

Tests in the TDS approach will fail initially. This is expected to verify their correctness once they pass after the feature is implemented. Security teams should help developers and operators implement controls in their software and infrastructure at first, taking each test one by one and providing guidance on implementation, and eventually transferring ownership of the tests to the DevOps teams. When a test passes, the teams are confident that the control has been implemented correctly and the test should never fail ever again.

An important part of TDS is to treat security as a feature of the product. This is achieved by implementing controls directly into the code or the systems of the product. Security teams that build security outside of the application and infrastructure will likely instigate a culture of distrust. You should shy away from this approach. Not only does it create tensions between teams, it also provides poor security as controls are not aware of the exact behavior of the application and miss things. A security strategy that isn’t owned by the engineering teams will not survive very long, and will slowly degrade over time. It is critical for the security team to define, implement and test, but it is equally critical to delegate ownership of key components to the right people.

TDS adopts the DevOps principles of automating the pipeline and working closely with teams. It forces security folks to build and test security controls within the environments adopted by developers and operators, instead of building their own separate security infrastructure. Covering the security basics via TDS significantly reduces the risk of a service getting breached, but does not remove the need for monitoring production environments. If this is all a bit much for you, consider consulting iam consultants for support and insight on IT security.

That’s all for this article.

For more on merging security with DevOps to help protect your organization and your customer’s data, check out the whole book on liveBook here and see this Slideshare Presentation.

[1]https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project