Kubernetes is a robust yet complex infrastructure system for container orchestration, with multiple components that must be adequately protected. In order to know how to more effectively secure your Kubernetes environments, it is important to understand the architecture of Kubernetes itself as well as where and how to focus efforts on valuable mitigations.
Each Kubernetes cluster consists of two sets of components:
The control plane, which is used to manage operations throughout the cluster, and
The cluster’s worker nodes, which run containerized applications in pods. Also, to achieve high availability and resiliency—especially for production cluster environments—both sets of components can be deployed across multiple machines that form a cluster.
Kubernetes control plane components
Considering that the Kubernetes control plane is designed to make global decisions regarding a cluster’s operations, compromise of control plane components could easily result in complete compromise of a cluster. Therefore, a critical starting point for any effort to secure Kubernetes is to initially focus on protecting its control plane services.
The Kubernetes control plane includes the components listed below, along with recommended steps you can take to increase security for each component.
Kubernetes API server (kube-apiserver)
This is the central control plane component that exposes the Kubernetes API and receives and validates application programming interface (API) requests for resource creation and modification. It serves as the main interface for cluster communications, since it is the only cluster component that all control plane and worker components can communicate with directly.
A number of vulnerabilities impacting the Kubernetes API server have been and continue to be disclosed. In addition to taking steps to mitigate these vulnerabilities by upgrading Kubernetes versions or applying security patches, restricting access to the Kubernetes API is key to securing clusters. This can be done by:
Setting up authentication for all Kubernetes API clients.
Configuring authorization that determines permissions regarding who can do what, using Kubernetes Role-based Access Control (RBAC).
Ensuring all API traffic is TLS-encrypted.
Audit logging in Kubernetes should also be enabled to ensure that API events are recorded for further analysis or investigation, as warranted.
This control plane component is the key value store that Kubernetes uses to store cluster data, including API objects and configurations.
Data stored in etcd needs to be adequately protected using encryption and access control since gaining visibility into this data can reveal significant details about your environment, including your containerized applications.
Kubernetes supports encryption at rest of sensitive data such as secrets, which should be enabled. Additionally, strong authentication and access controls must be enabled to limit both read and write access to the data store—write access to etcd grants full privileges on the entire cluster.
This is the control plane component that handles scheduling of pods on worker nodes based on resource availability, requirements and allocation, affinity/anti-affinity, taints/tolerations requests, and other policy constraints.
FIle permissions on the kube-scheduler pod specification and kubeconfig files should be restricted, and the service should be configured to only serve HTTPS. Additionally, the service should be bound to a localhost interface since it exposes health and metrics information which, by default, do not require authentication or encryption. Communication with the API server should be done using the secured port, in addition to requiring controls for authentication and authorization.
This control plane component runs processes that regulate, maintain, and adjust the state of things such as nodes, pod replicas, services, and services accounts and access tokens.
File permissions on the kube-controller-manager pod specification and kubeconfig files should be restricted, and the service should be configured to only serve HTTPS. It should be bound to a localhost interface as well. You should also ensure that an individual service account credential is configured per controller in conjunction with Kubernetes RBAC to ensure the control loops run with minimum required permissions. Communication with the API server should be done using the secured port, in addition to requiring controls for authentication and authorization.
This control plane component runs processes that manage cluster dependencies in cloud provider environments for functions such as node termination, route setup, and load balancing.
File permissions on the kube-controller-manager pod specification and kubeconfig files should be restricted, and the service should be configured to only serve HTTPS. You should also ensure that an individual service account credential is configured per controller in conjunction with Kubernetes RBAC to ensure control loops run with minimum required permissions. Communication with the API server should be implemented using the secured port, in addition to requiring controls for authentication and authorization.
Kubernetes node components
Each Kubernetes node runs the following components:
This node component is the primary node agent that manages individual containers running in pods and their state based on PodSpecs passed via the Kubernetes API server or as a file with the kubectl command line tool.
Vulnerabilities related to the kubelet have been and continue to be disclosed—these should be addressed by upgrading kubelet versions, applying available patches, and/or taking steps to mitigate the particular threat vector.
Limiting access to the kubelet with strong authentication and authorization is necessary because its HTTPS endpoints expose APIs that grant certain privileges on the node and containers. By default, access is unauthenticated.
This node component is a network proxy that handles request forwarding based on network rules and supports various network protocols (TCP, UDP, etc.). It enables Kubernetes services to be exposed externally.
If kube-proxy is using a file-based kubeconfig file, then file permissions should be restricted, Additionally, communication with the API server should be done using the secured port, in addition to requiring controls for authentication and authorization.
This component enables the functionality required to start, run, and manage containers on a given node, such as communicating with the kernel, setting up cgroups, etc. Some examples of runtimes supported by Kubernetes include Docker, CRI-O, and containerd.
The container runtime is what executes containers and, therefore, appropriate steps and security best practices should be taken to harden it. See the OpenShift Blog for recommendations on hardening Docker containers.
How Red Hat Advanced Cluster Security for Kubernetes helps secure Kubernetes components
By following the recommendations outlined above, you can help mitigate the majority of security issues that may arise with Kubernetes itself. However, it can also be challenging given the number of key Kubernetes components that exist and the various controls that should be implemented.
Red Hat Advanced Cluster Security provides a number of capabilities that automates DevSecOps for Kubernetes. It delivers comprehensive network visibility into control plane and node components, including communications between them and with additional services throughout the cluster. It identifies and prioritizes known vulnerabilities in the Kubernetes system itself to streamline remediation. It monitors permissions authorized via Kubernetes RBAC and applies built-in policies to surface violations related to the images or pod deployments for these components.
Finally, it enables runtime monitoring of system-level activity for the pods running in the kube-system namespace to detect anomalous and malicious events. If you are interested in learning more, request a demo.
A foundational aspect of securing your Kubernetes applications is to focus on securing the components that make up the Kubernetes infrastructure platform as a whole. This can be done by understanding those components and taking steps to protect them from vulnerabilities, misconfigurations, and runtime attacks. By doing so, you can immediately increase the security of your Kubernetes clusters and enable your organization to scale its use of Kubernetes in production with confidence.
About the author
Wei Lien Dang is Senior Director of Product and Marketing for Red Hat Advanced Cluster Security for Kubernetes. He was a co-founder at StackRox, which was acquired by Red Hat. Before his time at StackRox, Dang was Head of Product at CoreOS and held senior product management roles for security and cloud infrastructure at Amazon Web Services, Splunk, and Bracket Computing. He was also part of the investment team at the venture capital firm Andreessen Horowitz.