Red Hat SummitIl blog di Red Hat
DevSecOps is an approach to software development and deployment that takes full advantage of the agility and responsiveness of DevOps, but also makes security a shared responsibility that is integrated into the full life cycle of your apps.
As with most things, of course, there are some challenges to overcome when implementing DevSecOps in an organization. Effective DevSecOps requires more than new tools and mindsets, it involves cultural changes (and leadership buy-in) as well.
In an episode of the “In The Clouds” Red Hat Streaming series, Chris Short had Kirsten Newcomer on to talk about DevSecOps, discussing tools, culture and misconceptions. Here we’ve distilled that discussion into a Q&A in which Newcomer gives some tips and recommendations towards successfully implementing DevSecOps in your organization.
Newcomer is the Director for Security Strategy for the Hybrid Platforms Business Unit at Red Hat, with an emphasis on Red Hat OpenShift and Red Hat Advanced Cluster Security (formerly known as StackRox).
Please note that we have edited for brevity, flow, and clarity. This is not a verbatim transcript.
You can view the full original video on YouTube.
“Security isn't just for Ops teams anymore - what do we need to do to make security a focal point of app dev as well? And why is shifting security left important for containers and Kubernetes?”
Short: Security has classically been treated like it’s divorced from the software development process, and that it’s the “last checkbox” when deploying to production. So now the big question is: Why are we changing things now?
Newcomer: There are a couple of things around that.
I think you’re right -- security professionals have typically been siloed from the development teams in particular. They may be more connected to operations, but even there it’s been more about defining policies that the ops team has to implement as procedures and make sure they’re met.
In many ways DevOps is a follow-on to the agile development methodology, trying to apply some of those same principles not only to the development cycle, but also to getting that code released early and often.
Security was always intended to be part of that melding between development and operations, but it got overlooked in many cases when teams started doing DevOps. And even today, in some cases, DevOps is seen as something that belongs more to the application team, even though it’s intended to include ops.
So I tend to talk about “DevSecOps” because I want to explicitly call out that teams need to integrate security thinking into their DevOps life cycle.
Just as dev teams need to talk to ops to figure out -- in the DevOps process -- how to ensure they’re meeting the resiliency and regulatory requirements, they also need to be sure they’re meeting security requirements.
You mentioned how security often comes at the end, and there’s nothing worse for a development team than getting that security analysis and check at the point where they’re ready to deploy their code, only learning that they have to go back and start over. Nobody wants that.
One of the simple, useful things folks can do is do vulnerability analysis scanning in their continuous integration (CI) process, during their build process.
Even better is if you use something like the IDE plugins that Red Hat provides where you can do dependency analysis and get security data in your IDE in real time so that, even before you check in, you can learn whether there’s something that maybe needs to be fixed. But at minimum you should be doing this during your build process.
We always used to talk about the stats, about how much time does it take to fix a bug that’s found in production versus one that’s found in testing, and this is the same kind of thing. Let’s find and fix security issues as far left in the process as we can.
Short: Understanding the importance of having a software pipeline is huge -- it’s not just handing code off to the next team, you want to have some automation involved in that handoff.
Newcomer: Absolutely, especially when it comes to cloud native, but not just for cloud native.
When we think about containers and Kubernetes, a best practice is that you never patch a running container. And why is that? Because a container orchestration solution is designed so that if a running instance of your application goes down, a new instance will automatically be deployed from the container image. So if you’ve patched a running instance and not rebuilt your image, you lose that fix as soon as the redeploy happens.
You need your DevSecOps pipeline to have as much automation as possible so that you can respond quickly when new issues are found at whatever layer of your application they’re in.
You need automation for building, and for security testing, and also ideally, for quality engineering (QE) testing and user experience (UX) testing.
And that’s a challenge for some organizations right now -- how much test automation they can get in place. That can be easier for greenfield apps than it is for legacy apps.
Short: In your opinion, what is the right order of static analysis, dynamic analysis, and all that? Is there a preferred order of the way you do your DevSecOps, or is it really up to the team?
Newcomer: Some kinds of scans make more sense in different places. If you can find a tool that will give you dependency information in your IDE, that’s a big win, and same for vulnerability analysis.
But most vulnerability scanners -- which sometimes are also software composition analysis scanners -- they figure out what code is in the blob that you’re analyzing, and they tell you what those packages are, and then they map to known vulnerabilities. Those are best run in your container repository -- as soon as you pull it down, scan it.
And that includes content from Red Hat -- we’ll tell you if we know about vulnerabilities, but you should do the “belt and suspenders” thing and scan it anyway. Those scanners are typically good in the build process as well.
If you can do dependency analysis solutions like sneak in the IDE, that’s a great starting point. And then build vulnerability analysis into your CI process.
Any type of static analysis scanners have pros and cons. There are things like Coverity and other solutions that analyze your code for the quality, and to check if there are things like cross-site scripting risks in your code. You want to do those as early as possible as well.
Most of those scanners require some tuning, because you may wind up with false positives. And these types of scanners would ideally be in the CI process as well.
As for dynamic analysis, that will be in your test environment, so you’re doing that while you’re testing your application as it runs.
In any DevSecOps pipeline, you want to be sure you’ve got a test environment that’s as close as possible to your production environment so you can do your dynamic analysis in that test environment along with any automated user acceptance tests and things like that.
And again, this is one of the values of containers and Kubernetes is that you can do this. You can have an OpenShift cluster that looks just like your production cluster. Your development team, your user acceptance team, they can have the same environment with the same constraints. You’ve got all your system dependencies packaged with your container image, so this simplifies the debug process if something slips through testing and hits production.
When we talk “shift left,” and when we talk “DevSecOps,” we think about “DevSec” as in “what are the security tools that are useful to the developer?” And I think that’s because that’s the area that’s been neglected.
But DevSecOps also means “SecOps,” as in “what are the security tools that are useful in operations for containers and Kubernetes?” So you want runtime behavioral analysis, you want things like pod security policies, security context constraints -- admission control policies that help you control what privileges a container is given when it’s launched. And you also want processes and gates in place that ensure you’re only pulling approved images on to your production cluster.
So it really is the full life cycle that you want to be thinking about.
Short: Which brings us to the question of how do you bridge that DevSec and SecOps divide and have them work together in perfect harmony?
Newcomer: That’s a place where the software community is still working. It’s also one of the reasons I’m really excited about Red Hat’s acquisition of StackRox, because they have the ability to leverage data collected during the build process, and they can do vulnerability analysis and app config analysis, which is also important.
They can look at the metadata for your container and your pod, and check if they’re asking for privs, or if they have embedded secrets. That content can be analyzed and the data collected during the build process, and then it can be used by an admission controller that’s deployed on the cluster, and that’s where you get to say “OK, any pods that have these characteristics may not be deployed to my cluster.”
We really want tooling that leverages the data collected early, and if it wasn’t fixed, let’s make sure that informs our policy for deployment, and kind of have that full circle.
Short: So we know DevSecOps is important, and we know that we want to bridge the divide, but how do we bridge that divide in the cloud native and Kubernetes environments, because it’s so fast moving?
If you look at some of the reports from the last couple of years, you’ll see that the average lifetime of a container is measured in seconds. It’s almost like containers are reaching that serverless paradigm. So, how do we do that? What’s a good practice there for making that happen efficiently?
Newcomer: I think I have to answer both on the tooling and on the people side.
One of the things that’s really important to think about when it comes to cloud native workloads, is to assume that they are largely ephemeral. That means when you think about the traditional methods of securing your environment, you need to think about having both your platform and your applications -- your workloads -- as close to being born secure as possible.
You also need to have enough automation in place that when problems crop up -- because they will -- you are able to respond quickly. And that automation needs to be in place both for the application -- in the CI/CD pipeline for your app -- and for your platform. How are you managing deployment of your Kubernetes cluster, and are you prepared to redeploy at any moment should, for example, crypto miners be discovered on your platform?
You really want to take that DevOps approach that is understood to be useful for apps, and you want to do that for your platform too, so you want to think about it in a GitOps-like way. Everything should be “infrastructure as code,” and it should be monitored, assessed and evaluated. Then once it’s approved and signed off on, I should be able to deploy any time I want.
So that’s the tooling side, and there are a range of tools that can help you there. For the apps and the cloud native pipeline there are things like Jenkins and Tekton, for your operations there’s Ansible, Terraform, and all sorts of different things.
Argo CD is actually a really interesting new player that spans both of those. For a Kubernetes environment, leveraging Argo CD is a great way to go, too. But store your infrastructure -- treat your code, your configs, as if it were source code. Version it, manage it, automate all of that.
And then, on the people side -- back to your silo point -- people behave in the way they’re motivated. How do you help the security team that has been siloed to become partners in the process of building your DevOps pipeline for both apps and infrastructure?
One of my favourite customer examples is a team in the federal sector whose Chief of Cybersecurity decided that his team of security architects needed to learn the development process and something about the tools that the developers use. Once they learned those things, they could be much more effective advocates for helping get the process into a place that worked for everybody.
They really needed to shift their thinking, and that takes incentives. For an organization to make that kind of cultural change, you have to have somebody who’s willing to support that and help the team be rewarded for making those kinds of shifts.
Short: What tools would you recommend? What do you think are good tools to run against production, for example?
Newcomer: One of the things we do at Red Hat is work closely with our partners to ensure that the solutions are certified to run against OpenShift. That includes the ecosystem of security ISV partners, so when we think about it there are a lot of different categories to look at. One thing to consider is the value for your buck -- can you get a tool that helps you both with the pipeline and at runtime?
There are a number of our certified partners who do that in addition to StackRox -- which is being rebranded as Red Hat Advanced Cluster Security.
We also have partners like Sysdig, Palo Alto Networks’ Prisma Cloud Solution, Aqua Security -- and pretty much all of these partners have solutions that, at a minimum, you can use for vulnerability scanning in the pipeline as well as in a running cluster. And many of them have image assurance capabilities just like StackRox does.
And then you really want to be looking for runtime analysis capabilities -- behavioral runtime analysis is a great win. OpenShift, out of the box, uses security context constraints, so we have this built-in protection that prevents privileged containers from being deployed to worker nodes by default. But runtime behavioral analysis is a great addition for the production environment.
And then, are there solutions to help you do more with network controls? Kubernetes has network policies, but you have to figure out how to configure your environment appropriately for that.
There are solutions that can auto-suggest network policies for you. And that’s another way you can bridge the dev and the production environment -- if you’re running in a test cluster and you try out some of those suggested policies, that gives you an idea of what you actually want to deploy with.
Deep audit and monitoring is also key. Out of the box, OpenShift includes auditing on at the host level, API server auditing is on for Kubernetes, but many of our certified partner solutions also add deeper data collection and correlation.
Short: Is there anything you want to mention specifically about StackRox -- now called Red Hat Advanced Cluster Security -- that you think may set it apart?
Newcomer: One of the things I like about it is that it does that deeper data collection, which supports the auto suggest feature for network policies, and it also gives you a visualization of the network policy layer. Red Hat OpenShift Service Mesh visualizes Layer 7 with service mesh policies, but with StackRox you get that network policy layer visualization as well.
I’m really looking forward to what we’re going to be able to do with Service Mesh and StackRox together. We tick a lot of boxes, so when we think about that infinity arc that you see for DevOps or DevSecOps, we really feel like StackRox helps with that. We’re also really excited about the shift left elements, that we can do that early analysis and leverage that data for the admission control policies.
And we’re going to continue to work with all of our partners, so it’s not just about what we offer from Red Hat -- we genuinely do believe in choice, so look across what’s available and see what suits you best. Some of our customers have relationships with our partners, and they’re going to want to continue that.
We also really expect we’re going to get a lot of value from being able to tightly integrate the Red Hat ACS StackRox solution with OpenShift, because we’re now part of the same company.
Short: How can a developer get leadership buy-in?
Newcomer: It’s about the success of the business, so the business cares about getting capabilities, new solutions, differentiating innovative solutions out to the market as quickly as possible, and ensuring customer satisfaction along the way.
So when you think about meeting those goals, security has to be part of that, and the sooner that you can help the organization address their security concerns, the faster you can deliver, and the better you can address customer satisfaction.
Customers don’t want to have to update because a vulnerability that has been known for four years is present in something you just shipped. They understand if it’s a newly discovered vulnerability -- that happens, and everybody understands that -- but if the work just wasn’t done early to assess for vulnerabilities that have been around for a while, they’re going to be unhappy.
And we’ve seen breaches that have been due to situations like this, where there was a fix for the vulnerability that wasn’t applied. So, this is all about value to the business, value to the end user and the customer. If you can help your management understand that, as a developer, this is going to make your life easier and that you’re going to deliver more quickly if you have this ability -- that helps a lot.
So, you’re thinking in terms of not just technical, but a combination of technical and business value. Helping convince your management, then get your management enlisted to help convince other players that this is the right thing for the business.
And there’s another challenging scenario as well. A number of years ago I talked to a customer who was on the development team, and he would have loved to do vulnerability analysis early in his pipeline, but his security team was holding on tight. They didn’t want to give the AppDev team access to their security tools.
To deal with those silo scenarios, sometimes it requires executive support, but the best thing is to find a team that you can pilot these changes with, so they have an opportunity to iterate on the changes and things as you work together.
Ideally, you’d find somebody on that security team who’s willing to work with you, or maybe you have to bring somebody into the AppDev team who has a security background, and they can work with you to do a pilot, and you demonstrate the value through the pilot. And then you just advertise the heck out of it internally.
And I’d be remiss if I didn’t also mention that you should use some security language. Security teams think about “reducing risk,” so how can shifting security left reduce risk for the organization? We’re balancing business value against risk all the time, every day, and no CEO -- no CISO, no VP, nobody in your management chain -- wants their company splashed on the front page of the newspaper.
Short: So, yeah, use that as a bullet point, “we’re not going to be on the front page of the newspaper.” Put that right in your presentation, because that’s what’s going to get VPs attention.
Newcomer: We really had to think about that when we were re-crafting a bit of our language around how we go to market with our security story. Partly due to the StackRox acquisition, we wanted to use “build, deploy and run” as a key message, but while those words are great for a DevOps team, the security team is thinking, “How do I detect problems? How do I protect my environment? How do I respond to an incident?” And another way that team thinks is all about “risk management”. So absolutely, language is a key element.
Short: What would you say in the security realm keeps you up at night? From your perspective, what is the thing you worry about the most?
Newcomer: I assume that there’s going to be an unknown, that something’s going to come up that none of us predicted and it’s not so much the unpredicted that keeps me up at night, it’s more the organizations that haven’t yet found a way to update their processes to at least do the basics.
What really concerns me is that they’re not yet structured in a way to make it easy to update vulnerable packages or even be aware that they’ve got vulnerabilities in their environment. That they’re not yet doing the appropriate level of security for identity and access management.
So again, you kind of need that “belt and suspenders” approach. There’s always going to be something unexpected, so I really do worry most when people aren’t even using a belt.
Short: As far as misconceptions about DevSecOps, what do you think are some of the misconceptions people have about it?
Newcomer: First, that they think that they can just add a tool and it will fix everything. Even though I’ve talked a lot about vulnerability scanning, and we’ve talked about dependency analysis, tools aren’t enough. You also have to give your team time to respond to what those tools tell them.
I’ve seen organizations where developer productivity is measured by the number of lines of code they deliver, which is a terrible metric. Don’t measure your team on that, don’t measure them on the number of vulnerabilities that are found the first time you scan -- measure them on how quickly they can respond, or on the effectiveness of their tests, or whether they’re testing the right things.
You have to balance the speed of delivery with meeting the customer goals, and the customer goals include reasonable expectation of security. Balance those two and you’ve got to think about how you can help, because people respond to how they’re measured. What’s an effective measurement that will help your team do what you’re looking for?
Short: Do you think there’s a lack of qualified professionals in the DevSecOp arena?
Newcomer: Certainly it’s hard to find qualified security folks, and they’re expensive. So invest in growing them. Give people a chance to learn -- if you’ve got somebody on your AppDev team who’s interested in security, help them get training and they’ll be able to understand both sides of the puzzle.
Short: Let’s end on a high note. You work here at Red Hat as part of the Cloud Platforms Business Unit, so...what gets you up in the morning? What motivates you to come in and work for Red Hat every day?
Newcomer: I just love the opportunities in front of us, and the fact that we’re able to work with so many terrific companies and customers across such a wide variety of industries, which creates the opportunity to help them tackle really interesting problems that make a difference in the world.
Whether it’s health care, or energy delivery, or telco -- there’s just this really fascinating set of problems as we help our customers move into a more cloud native world. And we get to be there working with them on the forefront, figuring it out, and helping them on the journey. It’s a great deal of fun, and we have such a great group of people to work with here at Red Hat, too.
Learn more about DevSecOps
Sull'autore
Deb Richardson is the Senior Managing Editor for the Red Hat Blog.
