Security considerations are even more important today than they were in the past. Every day we discover new vulnerabilities that impact our computer systems, and every day our computer systems become more complex. With the deluge of vulnerabilities that threaten to swamp our security teams, the question, "How much does it matter?" comes quickly to our minds.
This question, "Does it matter?", has two parts:
How soon does it need to be fixed (a question for the software vendor)?
How soon does it need to be applied (a question for the end user)?
To set the stage, Red Hat Product Security uses the following terminology to help remove some of the ambiguity around discussing software vulnerabilities.
"Affected" — this is simply a yes or no decision to indicate whether the vulnerable code is present. It doesn't matter if the vulnerable code can't or won’t be executed. If it is present, then we consider the component to be affected.
"Severity" — this is a measurement of the issue’s criticality using a well-defined four point scale (Critical, Important, Moderate, Low). See the Red Hat security ratings page for more information about this, but be aware that — although the severity ratings will often be seen to track the corresponding CVSS score — they are not necessarily tied together.
We can use this JSON-schema library vulnerability as an example — CVE-2021-3918 The exact same vulnerability was assigned a different severity rating based on the product it was in. As a library by itself, Red Hat rated it as having an "Important" severity. However, on the first page of the affected packages table in the CVE page, we see that it has been assigned "Moderate" severity as embedded in Red Hat OpenShift Service Mesh.
In this blog, we’re looking specifically at containerized environments. Many of the expectations and assumptions for regular operating system environments simply do not hold true, and these differing factors come into play when assessing any given vulnerability. Like in the example above, containerized environments are different, and here we’ll examine some details on why and how this is the case. We show how much different the impact might be to a containerized environment, and good risk assessment can help provide an answer to the question: How important is this vulnerability?
What is a container?
Before we start analyzing vulnerabilities, we should define what a container is and why we use container images.
A container image is a unit that holds a set of software that packages up code and all its dependencies (application) that is ready to run. A well-crafted image will be lightweight and include only the necessary components to run a single application, including code, runtime, tools, libraries and settings. Although containers may be created that contain multiple applications, this is not common.
Well-designed container workloads look very different to traditional operating system environments in which multiple users share multiple applications on the same host. To better understand containerized environments like Red Hat OpenShift and how containers are used there please read: OpenShift overview.
Containerized environments are different
To illustrate the complexity of our computer systems, consider that in a typical OpenShift system, generally every Linux package exists in at least four places — on the underlying host system, within container images in the control plane, within container images in the platform's worker nodes and within container images in the user-provided workloads. Being able to accurately identify which part we are talking about is important, as is recognizing that any user-provided workloads are not considered when Red Hat does an analysis of the OpenShift platform itself.
Knowing this, we can start analyzing some of the types of vulnerabilities in containers.
In the three examples below, we look at some fairly typical types of vulnerability and explore in detail how their effect on a containerized environment can be evaluated.
In part, the Common Vulnerability Scoring System (CVSS) framework allows us to capture these details numerically in the "Environmental Score," although traditionally this is more often used by the end user of the software to capture the nature of the installation environment, not by the software vendor to capture alternative environments in which the software may be provided.
Privilege escalation vulnerability
Privilege escalation vulnerabilities are pretty common issues, but let’s try to understand how these types of vulnerabilities impact running containers. To demonstrate the impact assessment we will use one of the recent high-severity vulnerabilities: CVE-2021-4034
A detailed Red Hat security bulletin with details about this vulnerability can be found here: RHSB-2022-001 Polkit Privilege Escalation.
The vulnerability was found in
pkexec which is a part of the polkit package responsible for allowing unprivileged processes to communicate with privileged processes on a Linux system. Due to this bug, an unprivileged local attacker is able to escalate privileges bypassing any authentication and policies due to incorrect handling of the process’ argument vector.
Starting with the above information, we can identify that both "local attacker" and "escalate privileges" may have a different meaning inside a container. Further, we look at the specific containers in question and determine if a different analysis of the severity should be considered and if it can be justified.
The difficulty is that when we analyze a vulnerability as a whole, we don't know the details of any environment in which it may be present, and thus depend on a number of generalizations. In a containerized environment, and especially one where we are assessing a vulnerability present in an OpenShift component, we may actually know a great deal.
For the "local attacker" part of the vulnerability text, such a person can no longer be considered a low-privileged user of a shared system. They are at least a highly privileged user for any containers in question, and in the case of a control plane component, they may be presumed to already have some equivalency of cluster-admin permissions.
Next, we can look at the "escalate privileges" part. In a traditional operating system, such a privilege escalation would permit the attacker to compromise the host. In a container environment, it permits the attacker to compromise the container. It's slightly less clear-cut, but in the context of the container orchestration environment then CVSS' C:H (confidentiality: high) can be adjusted to C:L (confidentiality: low), and similar changes for integrity and availability are also made.
In summary, for a traditional operating system installation, this vulnerability can permit a low-privileged user to gain access to the entire system. As a Red Hat severity, we call this "Important". In an OpenShift environment, when we look at the containers identified as being provided by the platform itself, the vulnerability is limited to permitting highly privileged users to gain access to entire containers (that is, things they already have access to). As a Red Hat severity, we call this "Low".
Taking this process to the next step, we can see that a similar process of exploration can take place to examine a user-provided workload container in which the vulnerability is present. We can consider who the local users are and whether any exceptional circumstances are present, such as the containers running privileged and unconstrained and having access to the underlying host.
It should be noted that there also exists the possibility of a highly privileged container (for example something that runs as part of the control plane and leverages the "privileged" SCC to attain high levels of system access). In the case where a container like this is subject to an active vulnerability, we can see that exploitation of the container may be essentially equal to exploitation of the underlying host, and such a reassessment of the severity would not be suitable.
As you can see here, the same vulnerability that would warrant an urgent response for a regular operating system installation can be far less urgent for a containerized environment. There's no debate as to whether the vulnerability should be fixed (it should be), but having done a deeper analysis we can make the determination that it can be done in a measured fashion and not as a fire drill.
System crash vulnerability
Another common type of vulnerability is one that leads to a system crash. But how can we estimate the impact risk by these types of issues to containers? Of course, vulnerabilities might have different attack vectors and different impacts on various environments and systems, so let's examine one in a container environment.
Let’s use one of the recent well known vulnerabilities as an example: CVE-2021-33910
This flaw is related to the
systemd package and vulnerable
alloca function which allows a local attacker, who is able to mount a filesystem on a very long path, to crash the systemd service (PID 1) and, as a consequence, to crash the entire system.
systemd package is installed in many Red Hat containers, including the main Red Hat Universal Base Image (UBI). This means that this package exists in many Red Hat containerized products, such as OpenShift. This seems pretty scary, but in this case the key is how the vulnerable package is used in the system, which in this case is in the container.
In containers, PID 1 is not systemd (or initd in previous times) to manage processes for the entire host, but is the command or entrypoint that runs the specific application that's been containerized. The systemd binary may well be present in the container image but won't be running — any success an attacker may have in first starting it and then causing it to crash will have absolutely minimal impact on the container. In short, the vulnerability exists, but there's no reasonable method by which it can be exploited.
However, this doesn’t mean that our container is considered "not affected". As noted in the introduction, we use the word "affected" to state that the vulnerable code is actually present in the code — in this case the content of the container image. However, because the vulnerability is present but cannot be exploited, we assign it the lowest severity. As a matter of interest, Red Hat severity ratings do not go lower than "Low", but the CVSS rankings actually have "None" as an option for flaws with a CVSS of zero. In the CVSS base score this will be represented by having "C:N/I:N/A:N" — having a “None” rating for each confidentiality, integrity and availability.
Here ends our examination of the systemd vulnerability in the context of the OpenShift-provided container images. Remember that any user-provided workloads are outside the scope of the analysis, and in the event you've created "multiservice containers" that use systemd, a careful examination of the vulnerability in these containers should be undertaken.
Container escape vulnerability
Container escape vulnerabilities are very serious, and Red Hat Product Security spends a lot of time working to prevent them. Their impact significantly depends on the host (container environment) configuration, so let’s take a look at a recent Linux kernel vulnerability that was assigned an "Important" severity: CVE-2022-0185
The main consequence of this flaw is privilege escalation on the system. But there are posts on the Internet (for example CVE-2022-0185 in Linux Kernel Can Allow Container Escape in Kubernetes) which makes the claim that this flaw can be also used to achieve a container escape in Kubernetes.
Based on the described proof of concept, it appears that leveraging the local privilege escalation (exploit needs necessary privileges: CAP_SYS_ADMIN) is used to manipulate the root filesystem on the host. But, as was already said, the successful attack depends on the host configuration, like container runtime, system settings and any security mechanisms implemented on the host.
Red Hat Product Security analyzes every reported container escape vulnerability in many contexts to estimate the impact to all Red Hat products. For example, in OpenShift, the cluster processes on the node are namespaced, which means that switching in the running OpenShift container the namespace will not bring necessary full capabilities, even if these seem to be there. This means that the first step to achieve the container escape, i.e. leveraging the local privileges escalation, is already blocked because the necessary conditions to exploit this vulnerability are not provided. Additionally, a successful container escape attack is blocked by SELinux on the cluster node level. This means that for OpenShift, the impact of this vulnerability is reduced to “Low”.
This example shows how important a secure environment design is. With a good security controls design, vulnerabilities might be mitigated on many levels. For containerized environments, it means that potential attacks can be stopped at cluster level by applying good security practices (like correct RBAC,
restricted SCC by default) or on the container’s host level (SELinux, namespaces, seccomp). Following the recommended security practices can significantly reduce the potential impact of the vulnerability.
Mitigating affected packages
As we've seen in the above situations, a vulnerability’s impact on containerized environments can be different than on regular operating system environments. To estimate the impact of the specific vulnerability on your containerized environment, you must know how and why the vulnerable package has been included in the container image. For the custom deployments where, for example, the UBI image is used as the base image, you can rebuild the final container and remove the affected package during the build process. This action can be done if you can’t wait for the official patch or patched base image and you are sure that the affected package is not needed for your service.
In this article we used a couple of well known vulnerabilities to show that potential exploit impact might be different in various environments. Something that would be considered a “Critical” flaw for a standard physical or virtual operating system installation can have a “Low” impact for containers. It is very important to always remember about environmental factors that can reduce the impact significantly in many cases, and that Red Hat-provided container images may have different characteristics from your own.
It is imperative that any security or operations team have processes and procedures in place to respond to vulnerabilities of differing severity. These, along with an accurate assessment of any given vulnerability, allows us to treat high-severity vulnerabilities with the urgency they deserve for prompt attention, and allows us to treat low-severity vulnerabilities in a more measured manner.
Red Hat strives to publish accurate and timely information regarding our assessment of all vulnerabilities that are discovered in our products. We hope this article helps explain why our severity ratings will frequently differ for containerized environments, and further hope it gives you some tools and techniques you can use to examine your own containerized workloads to better assess the impact of any vulnerabilities that may be found in the packages or components that they contain.
About the authors
Przemysław Roguski is a Security Architect at Red Hat who specializes in Cloud Products security aspects. He contributes security analysis work on Red Hat OpenShift and other OpenShift-related products. He also designs security solutions and processes across Red Hat Product Security. He is focused on the security data improvements (various upstream and downstream security initiatives and projects like CWE, Kubernetes, Red Hat Vulnerability Scanner Certification program) to build better understanding of the security issues and improve client satisfaction.
Dave Baker has been with Red Hat since 2017. He's currently working as a Design Architect in the Secure Engineering team within Product Security, and has spent the last years in various security related roles helping to protect Red Hat OpenShift Container Platform and many other products.