Faster application development and release, quicker bug fixes, and increased feature velocity are three of the most often cited benefits of containerization. However, when security becomes an afterthought, you risk diminishing the greatest gain of containerization – agility.

Rolling out an application that hasn’t passed a security assessment puts the business at too great a risk. To prevent delays in application deployment and realize the benefits of containers and Kubernetes, organizations must shift left with security, building it into the development phase so they can address as many security challenges as possible during the build stage.

Earlier this month StackRox published the fourth edition of its State of Container and Kubernetes Security Fall 2020 survey report, where we examined how companies are adopting containers, Kubernetes, and cloud-native technologies and addressing their security challenges. Two of the findings include:

  • 90% of respondents have experienced a security incident in their container and Kubernetes environment in the last 12 months.

  • 44% of respondents said they’ve delayed application deployment due to security concerns.

In addition, we identified the most common types of security incidents reported by respondents and the security risks that they are concerned about the most. Taken together, organizations need to take the necessary steps to mitigate these risks so they don’t resort to delaying application deployment.

1: Misconfigurations and exposures

In our survey, respondents identified exposures due to misconfigurations as the most worrisome security risk in their container and Kubernetes environments, and for good reason: 67% of respondents have detected a serious misconfiguration in the last 12 months.

Configuration management poses a uniquely difficult challenge for security practitioners, particularly when using Kubernetes to orchestrate containerized apps. While a host of tools are available for vulnerability scanning of container images, configuration management requires more consideration. People know not to expose the Kubernetes dashboard to the Internet, but configuring a pod’s security context or implementing Kubernetes RBAC are just two examples of more challenging settings DevOps teams need to get right.

Pay close attention to how you’re configuring:

  • Container images - don’t use non-essential software (e.g, package managers, network tools and clients like curl, or Unix shells) that increases your security risk nor pull images from risky sources.

  • Secrets - don’t bake in secrets into images or expose them unnecessarily; use a secrets management tool like Kubernetes secrets and make sure deployments mount only the secrets they actually need.

  • Namespaces - use them, because they provide a key boundary for network policies and Kubernetes access control restrictions, and separating workloads into namespaces can help contain attacks and limit the impact of mistakes or destructive actions by authorized users.

  • Runtime privileges - follow best practices that adhere to the principle of least privilege.

  • Network policies - by default Kubernetes allows pods to talk to each other unimpeded. Network policies can be used as a key security control that prevents an attacker to move laterally through a container environment.

  • Persistent storage - make sure you have visibility into the use and configuration of persistent storage as this is a rare persistent vector in a mostly ephemeral container environment.

  • Control-plane - if you’re self-managing your Kubernetes clusters, then configuring the control plane components is critical because they make global decisions regarding a cluster’s operations, and compromise of any control plane component could easily result in a complete compromise of a cluster.

The best way to address these challenges is to automate configuration management as much as possible so that security tools – rather than humans – provide the guardrails that help developers and DevOps teams configure containers and Kubernetes more securely.

2: Vulnerabilities

We’ve seen several instances of serious vulnerabilities impacting containers and Kubernetes in the recent past. Common exploits of known vulnerabilities include crypto mining or other malware installation, and privilege escalation and host access. 

While image scanning at the build stage is a must, vulnerabilities pose a security risk to running deployments as well. Effective vulnerability management spans the entire container life cycle, and should:

  • Identify vulnerabilities in images, including in installed operating system packages and runtime libraries for programming languages.

  • Prevent images with vulnerabilities from getting pushed to production-accessible container registry.

  • Fail builds containing fixable vulnerabilities above a certain severity.

  • Use custom or third-party admission controllers in Kubernetes clusters to prevent the scheduling of vulnerable container images.

3: Runtime threats

The runtime phase is critical for container security because it presents a new set of security challenges. If you’ve shifted security to the left and minimized your security risk from vulnerabilities and misconfigurations, then the primary threat at runtime will likely come from external adversaries. There are a few things you can do here to mitigate your biggest security risks.

  • Monitor runtime activity - start with monitoring the most security-relevant container activities such as process activity and network communications among containerized services and between containerized services and external clients and servers.

  • Use the declarative data - use the build and deploy time information to evaluate observed versus expected activity during runtime in order to detect suspicious activity.

  • Limit unnecessary network communication - runtime is when you can see what kind of network traffic is allowed vs what’s actually needed for the application to function, giving you the opportunity to remove unnecessary connections.

  • Use process allow-lists - observe the application for a period of time to identify processes that are executed in the normal course of the application behavior, then use this list as your allow list against future application behavior.

4: Failed compliance audits

Compliance is one of the primary drivers behind container and Kubernetes security initiatives, and a failed compliance audit is usually due to security becoming an afterthought in the container adoption journey. There are several compliance standards specific to containers and Kubernetes that apply to all organizations, including CIS Benchmarks for Docker and Kubernetes, and NIST SP 800-190.

Industry-specific compliance standards include PCI-DSS, HIPAA, and SOC 2. A common mistake organizations make is to wait until they’re in production before considering their compliance requirements, or only focus on the runtime phase.

The best way to mitigate your risk from failing a compliance audit is to implement your security controls as early as possible both in your container adoption journey as well as the container life cycle. Automate your compliance checks and evidence reporting as much as possible to reduce overhead.

In order to reduce the security risks from containers and Kubernetes, companies first need visibility into their cloud-native environments. They need to understand how images are built and whether they contain any vulnerabilities, how the workloads and infrastructure are configured to operate, and where compliance gaps exist. 

With this information, Security and DevOps can begin to enforce policies to reduce the security risk to an acceptable level. 

Red Hat Advanced Cluster Security for Kubernetes can help you build, deploy and run cloud-native applications more securely. Learn more about Red Hat Advanced Cluster Security for Kubernetes through this datasheet or contact Red Hat to learn more.

Editor's note: This post was originally published on the StackRox blog.

About the author

Ajmal Kohgadai is Principal Product Marketing Manager for Red Hat Advanced Cluster Security for Kubernetes. Prior to its acquisition by Red Hat, he was the Director of Product Marketing and Growth at StackRox, a leading Kubernetes security company.

Read full bio