From the Application Security Program Handbook by Derek Fisher
This article discusses where security fits into the DevOps process along with some common approaches.
Read it if you want to learn more about baking security into DevOps.
What has been historically the case for software development and release is that there is a product team, a development team, a testing team, and an operational team that all take part in the delivery of features and products to a production environment. This is typically done in a way where each team has gates that start and finish their part of the process. DevOps tries to streamline that process and reduce or eliminate the hand-offs between teams. Most organizations will accomplish this by creating a single team that owns not just the development of the code, but the testing and delivery as well. In some smaller organizations there may only be a few people on the team that do all of these activities themselves, in larger organizations specialists would be used to perform specific tasks within the team. For instance, in these larger teams, there would be one or more people focused solely on the deployment, or just the testing, as well as having developers focused on the code. However, in all of these cases, the team would share the work items as one team.
DevOps is primarily a function of automation, tooling, and processes. This model works best when there is little to no manual work or processes. Some of the most successful organizations that use a DevOps model are able to deploy code from a development environment to a production environment in minutes. And they can do this multiple times a day. To achieve this level of speed and confidence, the organization will build a CI/CD pipeline.
The confidence in deployment is built in through strong controls around how source code is tracked, merged, and versioned. Testing must occur all throughout this model, from the developer’s unit test at the lowest level, up to the testing of API integration and system testing. Lastly, monitoring of the application in production for potential issues and having a rapid feedback loop to the development team will reduce the time to fix issues as they are found. This gives greater confidence to the development team that they can deploy and fix code rapidly. This certainly enables the developer to feel empowered, but where does security fit in?
DevOps supports security better
DevSecOps, SecDevOps, DevOps with Security, it’s been called many things. I’ve mostly heard it referred to as DevSecOps, so that is the term I will use here. Like the parent term of DevOps, the DevSecOps definition depends on the process and technology.
Many development methods provide a means for security to be integrated, however the DevOps methodology provides a unique opportunity for security to be more rapidly combined with other features in the development pipeline. As described, DevOps, and by extension DevSecOps, allows for development teams to have security fixes deployed to a production environment as quickly as the CI/CD pipeline can support.
In today’s environments, where your applications are running, security issues are ever present. In early 2021, Redscan released a report that reviewed the trends in vulnerabilities that were added to the National Vulnerability Database (NVD) from 1989 to 2020. This report highlighted how much has changed over that time and specifically how 2020 was a banner year for new vulnerabilities. The report found the following:
- More security vulnerabilities were disclosed in 2020 (18,103) than in any other year to date – at an average rate of 50 CVEs per day
- 57% of vulnerabilities in 2020 were classified as being ‘critical’ or ‘high severity’ (10,342)
- There were more high and critical severity vulnerabilities in 2020 than the total number of all vulnerabilities recorded in 2010 (4,639 including low, medium, high, and critical)
- Nearly 4,000 vulnerabilities disclosed in 2020 can be described as ‘worst of the worst’ – meeting the worst criteria in all NVD filter categories
- Low complexity CVEs are on the rise, representing 63% of vulnerabilities disclosed in 2020
What does this tell us about what we are up against in application security? Vulnerabilities are being released at a rapid pace and they are becoming easier to exploit. This means that in order stay ahead of significant issues, application security needs to be able to move swiftly as well. With this in mind, the methodology that the engineering organization has decided to use matters. For instance, in the Waterfall methodology, waiting possibly weeks or even months to release a resolution to a found vulnerability would leave the organization exposed much longer than necessary. Even in an Agile organization, a fix might be weeks away. Pretty fast, but not fast enough for an attacker that might already have code that exploits a CVE that was just publicly released. Attackers today can take information from a CVE and turn it into an exploit within hours.
This is where a methodology like DevSecOps can support the rapid deployment of security fixes. A well-tuned pipeline should allow, depending on the vulnerability, for a fix to a security issue to be deployed within hours assuming that the application security team has a well-defined process for triaging and assigning discovered vulnerabilities, and the DevSecOps team has the ability to pull in code changes, test, and deploy in a rapid manner.
Figure 1. DevSecOps process diagram
There are several key practices to keep in mind when building security into a DevOps pipeline.
- Security issues are no different than any other software defects. In fact, they should be indistinguishable from software defects considering that vulnerabilities are essentially a software defect that has security implications. Consider a defect like a memory leak in a software program. This can be leveraged by an attacker to consume the resources on the server and make the application unavailable. However, even without the security considerations, this is still an application issue that impacts the performance.
- Automation of security should be a priority. Just like there are automated tasks within the CI/CD that run tests, perform deployment, and run checks. Security should be no different. In this case, the running of Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST), Software Composition Analysis (SCA), infrastructure code configuration checks, and the like should be made part of the CI/CD with feedback to the development team.
- Once the previous step is integrated in order to ensure that security is an automated part of the CI/CD, the findings in those tools need to be presented to the development team as early as possible and enabled to block the build from completing successfully and deploying vulnerable code. This last point is critical. Code should not be allowed to continue to a production environment with known vulnerabilities. Just like the failure of a regression test would stop the build from progressing.
- Once the application is deployed to a production environment, it is important to continue to monitor it for security related issues. Things like insecure third-party libraries are of a constant concern and can be introduced after the code is deployed to production. If everything goes well in the CI/CD and there were no found vulnerabilities with the developed code, it still doesn’t mean that a third-party library that is used by the application would always be secure. Additionally, as infrastructure is today being setup and configured as code, called Infrastructure as Code (IaC), there is a unique opportunity to apply a consistent approach to secure infrastructure, but in order to do this the organization needs to have a method of monitoring for configuration drift. This is where a single change is made in a production environment, often through manual means, which varies from the IaC templates that the organization normally uses.
Considering these practices when developing the pipeline will give the organization the better protection against the deluge of vulnerabilities. Practices aren’t always enough. Having the right tools integrated at the right time matters as well.
Let’s imagine that there is a startup called Superior Products, and they have been using the DevOps methodology for some time. Their development teams consist of developers, testers, operational people, etc. And they have worked tirelessly over the past few months building in the following tools and processes:
- SAST (static application security testing) has been integrated into the developers IDE (integrated development environment) where the developer has the opportunity to scan their code for security issues early.
- SCA (software composition analysis) is integrated into the IDE as well as in each developer’s browser as a plugin. The IDE integration allows for the developer to identify issues in the third-party libraries that they are using in the building of their software. The browser plugin allows the developer to research new libraries with confidence that there are no known security vulnerabilities impacting the library they are reviewing
- When the developer issues a pull request to submit code, they code repository tool requires the developer to submit the results of their SCA and SAST in the pull request.
- The Continuous Integration tools have SAST and SCA tools enabled that incrementally scan the code changed when the developer’s code is integrated with additional code changes from other developers.
- DAST (dynamic analysis security testing) is integrated into the testing environment to ensure that once the code is integrated and deployed to the testing environment, security test used to scan while the application is running are triggered. This also includes the use of automated penetration testing tools to augment the DAST scan.
- They have integrated the ticketing system that is used in Superior Products so that any tools that identifies a security issue will generate a defect, assign it to the appropriate development team, and alert them of a new finding on their collaboration channels. The defect is opened with recommendations from the scanning tools on how to resolve the found issue.
This setup allows Superior Products to rapidly identify security issues, get them to the right team, provide resources on how to resolve, a criticality and timeline for resolution based on the organizations resolution policy. This well-oiled machine is capable of finding and resolving security issues rapidly before the code is deployed to production.
Now let’s imagine that Superior Products have a competitor called Acme Services. They’re not as up to speed as Superior Products and although they are using an Agile Methodology their releases are only deployed to production once every four weeks. They are still relying on a SAST tool that only runs once the code has been checked into their code repository and the continuous integration engine picks up the changes. Furthermore, the development team has decided not to trigger a failed build when vulnerabilities are found due to the timeline commitments to get code deployed at the end of the four-week sprint. This means that code is built with vulnerabilities identified and is then deployed to a testing environment with the known vulnerabilities. Acme Services has decided to partner with their internal penetration testing team to run a penetration test prior to the final build package being completed. This ultimately means that vulnerabilities are discovered late with the intention of queuing up the vulnerabilities that are found into the next sprint and subsequent release. This leaves them potentially exposed for weeks and shows the power of having tools integrated throughout the process.
Clearly some of the reason that Superior Products is doing better than Acme Services is due to their adoption of DevSecOps, which gives them the ability to react to problems more quickly than their competition. This allows their developers to spend more time on features and actual development than putting out proverbial fires.
If you want to learn more, check out the book on Manning’s liveBook platform here.