Security concerns remain the number one challenge for adopting and running containerized applications in Kubernetes. Red Hat’s State of Kubernetes Security Report, which analyzed survey data from over 500 IT and security decision-makers, discovered a similar trend. The report revealed that 59% of respondents are most worried about unaddressed security and compliance needs or threats to containers.
Organizations are eagerly adopting containers and Kubernetes. If they do not make the necessary investments in security strategies and tooling simultaneously, they risk the security of their critical applications and may need to delay application rollout. In this blog post, we examine the most common security issues that organizations are facing and how to address them.
Look out for these common Kubernetes security issues
A whopping 94% of survey respondents have experienced a security incident in their Kubernetes and container environments during the last 12 months.
Kubernetes and containers, while powerful, increase this risk substantially due to a lack of existing security skills and knowledge and insufficient tooling. Kubernetes has powerful functionality applied through a declarative model. A single workload may require significant configuration to ensure a more secure and scalable application. Add on technical debt and organizational hurdles, and it is a challenge even for experienced Kubernetes professionals to get everything right all the time.
Human error is the most often cited cause of data breaches and hacks. The recent news of misconfigured Microsoft Power Apps portals exposing millions of highly sensitive health records, including Covid vaccine data, is the latest reminder of what happens when you combine developer-friendly (insecure) default configurations with human oversight.
Furthermore, configuration management poses a uniquely difficult challenge for security practitioners in Kubernetes. While many tools are available for vulnerability scanning of container images, configuration management requires more consideration. People may know that they should avoid deploying the Kubernetes dashboard, but configuring a pod’s security context or implementing Kubernetes RBAC are just two examples of more challenging settings that teams need to get right.
Not surprisingly, nearly 60% of respondents have experienced a misconfiguration incident in their environments over the last 12 months. Nearly a third have discovered a major vulnerability, and another third said they have suffered a runtime security incident. Lastly, 20% said they had failed an audit.
At the same time, worries over misconfigurations lead the list of security concerns for respondents (47%) – almost four times the level of concern over attacks (13%), with vulnerabilities as the second leading cause of worry (31%).
Minimizing Kubernetes misconfigurations
The configuration options for container and Kubernetes environments run deep and can be challenging to get right. In large environments, it is nearly impossible to manually check each security configuration for each asset to assess its risk. At the very least, avoid using default configuration until you understand its security implications and it does not exceed your risk tolerance while keeping in mind that Kubernetes defaults are frequently inherently open and insecure.
The CIS Benchmarks for Kubernetes provide helpful guidance and a useful framework for hardening your environment. They contain hundreds of checks for different configuration settings. Ensuring continuous adherence to the CIS benchmarks and other configuration best practices requires a certain level of automation to ensure consistency.
As a best practice, create risk-based policies that are integrated into your CI/CD pipeline. This way, you can halt commits or builds that do not meet your minimum security requirements and provide guardrails for developers within the same tools they are most familiar with. Use a dynamic admission controller in Kubernetes to prevent misconfigured applications posing a high-security risk (such as containers running as root with unrestricted network access) from being deployed into production environments.
Protecting against runtime threats
When you deploy containerized images into production Kubernetes clusters, they face new security issues and challenges along with external adversaries. The goal at runtime is to detect and respond to anomalous application behavior that might indicate a malicious actor is attempting to:
- Breach your environment via
- Stolen credentials
- Comprised images in the registry
- Application vulnerabilities
- Exposed dashboard
- Kubeconfig file
- Establish persistence
- Gain additional access through lateral movement, privilege escalation, or credential access Execute malicious code
- Hijack or destroy resources
Protecting Kubernetes clusters and workloads at runtime requires a multipronged approach using defense-in-depth. You should combine automated process discovery and behavioral baselining with processes allowlists to identify anomalous behavior. At the same time, you should use pre-defined threat profiles that look for common threats, including cryptocurrency mining and privilege escalations to quickly identify and respond to such threats in real-time.
Detecting and remediating risky vulnerabilities
One of the most critical steps in securing containers and Kubernetes is to prevent images or containers with known vulnerabilities from being deployed and to identify, triage, and stop running containers that have vulnerabilities. Image scanning provides one of the earliest opportunities to lower your security risk, especially when integrated early into a CI/CD pipeline.
A comprehensive vulnerability detection solution should identify vulnerabilities based on specific languages packages and by each container image layer. Additionally, you should run on-demand vulnerability scans to discover vulnerabilities across images in running deployments, ensuring that newly discovered vulnerabilities are promptly detected. Adding Kubernetes context into vulnerability scans such as pods, namespaces, deployments, and clusters will enable development teams to understand the impact for insecure containers. The combination of declarative configuration data from Kubernetes with vulnerability data can help you prioritize remediation efforts.
Passing compliance audits
Most industry compliance standards, such as HIPAA or PCI-DSS, provide a broad framework of what is required to pass a given check but leave it up to the organization to determine the right controls. The Center for Internet Security (CIS) offers its own standards (CIS Benchmarks for Kubernetes) to help organizations follow security best practices.
If you adhere to the CIS Benchmarks, you will have an advantage in achieving compliance for other industry regulations. That is because some of the core security standards (such as networking and workload isolation, vulnerability detection, access control, and data protection using cryptography) espoused in industry regulations are also covered in CIS Benchmarks.
Passing a security audit does not always go smoothly and can be costly the longer they persist. If organizations follow the security best practices outlined in the CIS Benchmarks, by ensuring that your workloads and infrastructure are protected against high-risk vulnerabiltiies and misconfigurations, following a DevSecOps model, and implementing runtime security, the next compliance audit might be less painful.
About the author
Ajmal Kohgadai is Principal Product Marketing Manager for Red Hat Advanced Cluster Security for Kubernetes. Prior to its acquisition by Red Hat, he was the Director of Product Marketing and Growth at StackRox, a leading Kubernetes security company.