Introduction

As part of my exploration of Kubernetes, while working on a project I wanted to execute commands inside a pod. Rather then forcing the container to have some specific behaviour, I wanted to utilize the API mechanism exposed as the kubectl exec subcommand. While investigating, I found that exec doesn’t yet sport extensive documentation, and hopefully this post will help those who find themselves in a similar situation.

API endpoint

The Kubernetes docs do not mention the exec endpoint, but OpenShift’s documentation does offer some basic information. From that, we know where the endpoint lives and what parameters we need to pass.

We need to issue POST requests to this path.

/api/v1/namespaces/$NAMESPACE/pods/$NAME/exec

the two strings that need to replaced in the path are fairly obvious, and the query string parameters are described correctly in the table in the OpenShift documentation, with a single exception: The command parameter can be included multiple times.

First, let’s take a look at what it looks like for a single command parameter, one that will simply execute /bin/bash in the pod:

/api/v1/namespaces/project-1/pods/pod-1-lmlzj/exec?command=/bin/bash&stdin=true&stderr=true&stdout=true&tty=true

Now for multiple command parameters:

/api/v1/namespaces/project-1/pods/pod-1-lmlzj/exec?command=/bin/bash&command=-c&command=/bin/bash&stdin=true&stderr=true&stdout=true&tty=true

which gives us something like [‘/bin/bash’, ‘-c’, ‘/bin/bash’] which could be logically transcribed as /bin/bash -c “/bin/bash”.

Protocol

kubectl and oc use the SPDY protocol at the moment, which is being deprecated . The second option is to use Websockets, which seems to be the best way. Anyway one of these two protocols, SPDY or WebSockets, is required for communication with this endpoint, and the API will refuse requests without Upgrade headers.

HTTP headers

To provide all the necessary information, the request needs to contain the set of headers required by the API. Some of them will be handled by your WebSockets client (e.g. Upgrade, etc.), but there are two that need to be provided by the user.

The first one is Authorization, with a value of Bearer <token> that authenticates the request. For Kubernetes, follow this guide. With OpenShift, simply get the token for your user:

oc whoami -t

The other header is Accept, with the value */*. Any other value will be rejected with 406 Not Acceptable, even though the example shown above in the documentation shows the incorrect value of application/json. There is an issue in progress to make the documentation accurate.

Communication protocol

With all the information in place, the WebSocket should be able to establish a connection and the API will start communicating. When you write to the WebSocket, the data will be passed to standard input (stdin) and on the receiving end of the WebSocket will be standard output (stdout) and error (stderr). The API defines a simple protocol to multiplex stdout and stderr over a single connection. Every message passed through the web socket is prefixed by a single byte that defines which stream the message belongs to.

|Code|Meaning |

|----|--------|

|0   | stdin  |

|1   | stdout |

|2   | stderr |

So for every message received over the socket, you need to get the first byte and decide whether it is stdout or stderr. In Ruby, this would look something like:

data = [1, 27, 91, 63, 49, 48, 51, 52, 104, 98, 97, 115, 104, 45, 52, 46, 50, 36, 32]
case data.shift
 when 1
 $stdout << data.pack('C*').force_encoding('utf-8')
 when 2
 $sterr << data.pack('C*').force_encoding('utf-8')
 else
 unknown_data(data)
 end
 

To send data to the API, you need to convert to bytes and prepend 0 to indicate the message belongs in the stdin stream:

data = ‘ls -la\n’

data = data.unpack(‘C*’) # [108, 115, 32, 45, 108, 97, 13]

socket.send(data.unshift(0))

Connection lifecycle

One last problem is that there may be proxies and other “roadblocks” on the way to the API, or you may simply reach the TCP timeout. To get around that, send an empty message every once in a while to keep the connection busy:

Thread.new

  loop do

    socket.send([0])

    sleep(30)

  end

end

Conclusion

With this information, it should be possible to write your own application to communicate through the Kubernetes API with processes running inside your Kubernetes clusters. The sample Ruby excerpts have been tested on OpenShift 3.7.1, using minishift.

$ oc version

openshift v3.7.1+282e43f-42

kubernetes v1.7.6+a08f5eeb62

While the examples use Ruby, it should be straightforward to translate them into your favourite language.

If you can read Go, you can check how the endpoint is used by kubectl itself in the [upstream source code] (https://github.com/kubernetes/kubernetes/blob/release-1.7/pkg/kubectl/c…).