Understanding the critical Kubernetes privilege escalation flaw in OpenShift 3

A critical security vulnerability in Kubernetes (CVE-2018-1002105) affecting OpenShift Container Platform 3 was patched by Red Hat in December 2018. The OpenShift API Server is a component which handles all the API requests for OpenShift Container Platform 3. The API Server had a vulnerability in the ‘upgradeawarehandler’ type which acts as a reverse proxy to the compute nodes and other services added as API Extensions. A flaw in the reverse proxy prevented it from closing connections from the downstream service when there was an error. This allowed the user making the API call to escalate their privileges to those of the API Server. For details of the patch, see the upstream commit for Kubernetes.

Context

The OpenShift API Server is included in all OpenShift 3 installations and handles all the administration tasks for the cluster. Administrators and developers don’t usually call the API directly, but use the ‘oc’ binary, which calls the API server. Further information including detection, mitigation, and patching instructions can be found in the vulnerability article.

Amongst other things, the API server provides the functionality for the following oc commands:

• oc exec

• oc port-forward

• oc rsh

It does this by acting as a reverse proxy to the kubelet running on the compute nodes. When connecting to the kubelet in order to fulfill any of the above commands it opens a websocket connection which connects stdin, stdout, or stderr to the administrator or developers original call. For more information, read the Remote Commands section of the OpenShift Container Platform 3.11 architecture guide.

The API Server also acts as a reverse proxy when implementing the API Aggregation feature of Kubernetes. API aggregation enables the installation of additional Application Programming Interfaces (APIs) into the core API Server. Those additional APIs are referred to as API extensions by upstream Kubernetes. API Extensions allow an architect to extend the features of OpenShift Container Platform 3.

Exploiting pod attach/exec/port-forward

In order to use pod attach/exec/port-forward permissions to exploit this vulnerability in OpenShift Container Platform 3, an attacker would need to have the permission granted by the ‘admin’, or ‘edit’ OpenShift roles normally allocated to developers.

When connecting to the kubelet, the API Server uses certificate based authentication. The certificate used to connect to the kubelet grants it ‘cluster-admin’ privileges, which allows full access to the kubelet. Authorization of the actions allowed by an attacker are only checked when the request is received by the API Server. If an attacker sends a request which is authorized, such as exec to their own pod, it is passed onto the kubelet.

However, if an attacker sends a request which includes a Connection Upgrade HTTP Header and is denied, the kubelet returns a non HTTP 101 status response, but the connection between an attacker and the kubelet is not closed. This results in the attacker being able to send subsequent requests to the kubelet without the API Server checking if the attacker is authorized to perform those requests. The patch rectifies the situation by checking the response from the downstream service (kubelet in this case) and closes the connection made by a potential attacker to the API Server.

If there are any privileged containers running on the compute node alongside a pod which an attacker has permissions to run remote commands on, they can send a subsequent request to the kubelet via the API Server’s open websocket to the privileged container instead. For example in OpenShift 3.11 the ‘ovs’ container in the ‘openshift-sdn’ namespace runs as a privileged container on every node. Getting access to this container allows an attacker to remount the host filesystem.

Exploiting API Extensions

By default the Service Catalog API Extension is installed in OpenShift Container Platform from versions 3.7 to 3.11. It was a Tech Preview feature in 3.6 and was not installed by default. A user can check if they have it installed in the cluster by issuing an API request to the ‘servicecatalog.k8s.io’ endpoint. The following instructions show how to do that by using the ‘oc proxy’ command, which sets a reverse proxy to forward request from port 8001 on the loopback device locally to an API Server which you are currently logged in.

$ oc proxy

Starting to serve on 127.0.0.1:8001

Then in a new terminal window:

$ curl http://localhost:8001/apis/servicecatalog.k8s.io/v1beta1/

{

 "kind": "APIResourceList",

 "apiVersion": "v1",

 "groupVersion": "servicecatalog.k8s.io/v1beta1",

 "resources": [ 

{

  "name": "clusterservicebrokers",

  "singularName": "",

  "namespaced": false,

  "kind": "ClusterServiceBroker",

  "verbs": [

    "create",

    "delete",

    "deletecollection",

    "get",

    "list",

    "patch",

    "update",

    "watch"

  ]

},...

While ‘oc proxy’ commands add authentication options to the API call forwarded to the downstream service, the authentication options are not required to access that endpoint. This can be tested by changing the hostname from the proxy server URL to that of the OpenShift web console.

While an unauthenticated user can send requests to this service catalog discovery endpoint, they should not be able to invoke any of the API calls listed as resources by the HTTP response to that call, for example, http://localhost:8001/apis/servicecatalog.k8s.io/v1beta1/clusterservice…

Using this vulnerability exposed by the API Server, an attacker is able to create a malicious request to the service catalog which returns an error from the downstream service catalog to the API Server. An attacker can then use the unclosed HTTP connection, to send a subsequent request directly to the service catalog, and can now add X-Remote-User and X-Remote-Group headers to fake any user or group, including cluster-admin.

The extension-apiserver-authentication-reader role allows an API Extension to take part in the TLS certificate authentication chain. That is, it will be able to read the extension-apiserver-authentication configmap in the kube-system namespace which contains a CA cert. Any client key signed by this CA is trusted for authentication via user and group HTTP headers. The API Server sends the authenticated user and groups via those headers and the API extension will know that these values are trustworthy and will use them to do delegated authorization. Controlling these headers therefore allows arbitrary privilege escalation.

Knowing that the API extension needs the extension-apiserver-authentication-reader role for this means that we can query the cluster for service accounts which have a binding to that role.

​$ oc get rolebindings -n kube-system -o=custom-columns=ROLE:.roleRef.name,USERNAME:.userNames | grep extension-apiserver-authentication-reader 

If there is any service account with a role binding linking it to the extension-apiserver-authentication-reader role, that API extension could potentially be used to exploit this vulnerability. We have found the service catalog and metrics service API Extensions to be vulnerable, however your cluster may have another custom API extension making use of this aggregated API server feature. Also, we have found that Openshift template service broker is not affected, so it’s not necessary to disable it.

If we know a target API extension exists on the cluster, we can issue a malicious request via the API Server which will cause the downstream API extension to return an error to the API Server. However, as with the pod exec attack, the HTTP connection will remain open for the attack to send malicious requests directly to the API extension via an open websocket. Those requests include a client key, signed by the trusted CA cert, and the request will be allowed by the API extension. This allows an attacker to make any API calls to the downstream service. In the case of the service catalog, that allows an attacker to create a service broker in any namespace, and on any node.

Call to Action

If you are running an OpenShift Container Platform 3 cluster, apply the patches listed in the vulnerability article as soon as possible. If the patches cannot be applied immediately, consider applying both of the mitigations offered. If you trust the users of your cluster with pod attach/exec/port-forward permissions, it may only be necessary to disable any API extensions installed on your cluster until you can apply the full patch.

Thanks goes to the Kubernetes Security team for reporting this issue to Red Hat Product Security under embargo. Also, I would like to thank Red Hatters David Eads and Stefan Schimanski for their help writing this article, as well as Red Hat’s Paul Harvey and Sam Fowler for their help investigating the vulnerability and its impact.