The shift to rapid, frequent deployments over the past decade initially left application security behind. Companies began shipping innovation to their customers faster and faster, with releases moving from quarterly to multiple times per day. The application security processes of old, with a heavy focus on penetration testing, couldn’t keep up. With the rapid rise of DevOps, security teams became either a blocker to production release, or were in a position of constant catch up.
In recent years, however, security is finally catching up. Leading companies are now integrating security testing into CI/CD, running automated tests with every PR (or even every commit), and catching bugs long before they hit production.
The New Model: Automated AppSec Testing in CI
We have a proven model for application testing. Today’s engineers run unit tests and integration tests as part of building software. Security testing should be the same.
For quite a while, we lagged as an industry, leveraging infrequent tests of the customer-facing production applications of our means of ensuring security. Security is finally making the shift into developer workflows and CI/CD pipelines.
And that is clearly the place it should live.
You can think of security testing as negative integration testing. As opposed to traditional integration testing that ensures everything works correctly at runtime, you are testing that nothing works *incorrectly*. The focus of this integration testing just happens to be security.
With automated application security checks in the DevOps pipeline, you break the build if a new potential vulnerability is found. And if nothing new is introduced, you can rest easy knowing that you are testing your application’s security.
Your testing should leverage both automated scanning technologies as well as custom scripts for your specific application (think tenancy checks, for example). There are open source tools that can help in this process, such as Zed Attack Proxy (ZAP), or SaaS providers, such as StackHawk (brief shameless self promotion there…).
Why Security Test in CI/CD?
Finding Bugs in Prod is Too Late
Some engineering teams still rely on scheduled scans of production or quarterly pen tests as their means of security testing. Hopefully this is stating the obvious here, but one big problem with scheduled scans of production (or even worse, the pen test) is that THE VULNERABILITY IS IN PRODUCTION!
Not only is this a problem because you have an exploitable security bug available to attackers until your next test, but you also create one of two situations when a bug is found. Either you have an all-hands-on-deck firefight (which is bad for culture and pulls people away from planned work) or the bug gets prioritized in upcoming work and remains exploitable for an even longer period.
Obviously, this approach is questionable at best.
Smaller Units of Change, *Much* Faster Fix Times
Raise your hand if you have been part of an engineering team that has received a 50 page penetration test report (typically in PDF) and been given the mandate to go fix the security gaps. Inefficient doesn’t begin to describe this process.
When you get your quarterly pen test, figuring out where a particular finding lives in code takes a long time. You likely have to dust off a part of the code base that you haven’t touched for months. When companies test with every PR, engineers typically know exactly where they went wrong, can quickly fix the bug, and can quickly update the PR. At a minimum, they know where to start the investigation.
Scan Microservices, Not the End State Application
When you scan app.acmecorp.com and find a security bug, figuring out how to fix that bug can be a heavy lift. Not only does your team lack the context of what it was working on, but figuring out which underlying service the bug actually lives in can be a tall task in and of itself. As a result, getting to a fix can take a long time and can be subject to internal ticket volleyball, where the work is being passed from team to team.
Instead, application security tests should be run in CI on the underlying microservices. In doing so, you limit the footprint of where the potential vulnerability lives in code, and your fix is likely aligned with the team that is familiar with and working on that service.
Again, this gets you faster fixes and gets you back to building software that is valuable to your customers.
Get Started (…it is easier than you think)
I have been in many conversations with well intentioned engineering teams that want to add application security testing into their CI pipeline, but haven’t moved on it because it feels like such a heavy lift. More and more companies are adding automated security testing into CI, and getting started is easier than you think. Here is always my recommendation:
Pick your tooling (I’m biased here, but I obviously recommend StackHawk)
Run your initial test and put everything into a backlog for prioritization
Automate in CI to break the build on any *new* issues
With this approach, you can start chipping away at any security gaps that do exist, while ensuring that nothing net new is introduced.
