Introduction
As part of my exploration of Kubernetes, while working on a project I wanted to execute commands inside a pod. Rather then forcing the container to have some specific behaviour, I wanted to utilize the API mechanism exposed as the kubectl exec
subcommand. While investigating, I found that exec
doesn’t yet sport extensive documentation, and hopefully this post will help those who find themselves in a similar situation.
API endpoint
The Kubernetes docs do not mention the exec
endpoint, but OpenShift’s documentation does offer some basic information. From that, we know where the endpoint lives and what parameters we need to pass.
We need to issue POST requests to this path.
/api/v1/namespaces/$NAMESPACE/pods/$NAME/exec
the two strings that need to replaced in the path are fairly obvious, and the query string parameters are described correctly in the table in the OpenShift documentation, with a single exception: The command
parameter can be included multiple times.
First, let’s take a look at what it looks like for a single command
parameter, one that will simply execute /bin/bash
in the pod:
/api/v1/namespaces/project-1/pods/pod-1-lmlzj/exec?command=/bin/bash&stdin=true&stderr=true&stdout=true&tty=true
Now for multiple command parameters:
/api/v1/namespaces/project-1/pods/pod-1-lmlzj/exec?command=/bin/bash&command=-c&command=/bin/bash&stdin=true&stderr=true&stdout=true&tty=true
which gives us something like [‘/bin/bash’, ‘-c’, ‘/bin/bash’] which could be logically transcribed as /bin/bash -c “/bin/bash”
.
Protocol
kubectl
and oc
use the SPDY
protocol at the moment, which is being deprecated . The second option is to use Websockets, which seems to be the best way. Anyway one of these two protocols, SPDY or WebSockets, is required for communication with this endpoint, and the API will refuse requests without Upgrade
headers.
HTTP headers
To provide all the necessary information, the request needs to contain the set of headers required by the API. Some of them will be handled by your WebSockets client (e.g. Upgrade
, etc.), but there are two that need to be provided by the user.
The first one is Authorization
, with a value of Bearer <token>
that authenticates the request. For Kubernetes, follow this guide. With OpenShift, simply get the token for your user:
oc whoami -t
The other header is Accept
, with the value */*
. Any other value will be rejected with 406 Not Acceptable
, even though the example shown above in the documentation shows the incorrect value of application/json
. There is an issue in progress to make the documentation accurate.
Communication protocol
With all the information in place, the WebSocket should be able to establish a connection and the API will start communicating. When you write to the WebSocket, the data will be passed to standard input (stdin
) and on the receiving end of the WebSocket will be standard output (stdout
) and error (stderr
). The API defines a simple protocol to multiplex stdout
and stderr
over a single connection. Every message passed through the web socket is prefixed by a single byte that defines which stream the message belongs to.
|Code|Meaning |
|----|--------|
|0 | stdin |
|1 | stdout |
|2 | stderr |
So for every message received over the socket, you need to get the first byte and decide whether it is stdout
or stderr
. In Ruby, this would look something like:
data = [1, 27, 91, 63, 49, 48, 51, 52, 104, 98, 97, 115, 104, 45, 52, 46, 50, 36, 32]
case data.shift
when 1
$stdout << data.pack('C*').force_encoding('utf-8')
when 2
$sterr << data.pack('C*').force_encoding('utf-8')
else
unknown_data(data)
end
To send data to the API, you need to convert to bytes and prepend 0
to indicate the message belongs in the stdin
stream:
data = ‘ls -la\n’
data = data.unpack(‘C*’) # [108, 115, 32, 45, 108, 97, 13]
socket.send(data.unshift(0))
Connection lifecycle
One last problem is that there may be proxies and other “roadblocks” on the way to the API, or you may simply reach the TCP timeout. To get around that, send an empty message every once in a while to keep the connection busy:
Thread.new
loop do
socket.send([0])
sleep(30)
end
end
Conclusion
With this information, it should be possible to write your own application to communicate through the Kubernetes API with processes running inside your Kubernetes clusters. The sample Ruby excerpts have been tested on OpenShift 3.7.1, using minishift.
$ oc version
openshift v3.7.1+282e43f-42
kubernetes v1.7.6+a08f5eeb62
While the examples use Ruby, it should be straightforward to translate them into your favourite language.
If you can read Go, you can check how the endpoint is used by kubectl
itself in the [upstream source code] (https://github.com/kubernetes/kubernetes/blob/release-1.7/pkg/kubectl/c…).
About the author
Browse by channel
Automation
The latest on IT automation for tech, teams, and environments
Artificial intelligence
Updates on the platforms that free customers to run AI workloads anywhere
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
The latest on how we reduce risks across environments and technologies
Edge computing
Updates on the platforms that simplify operations at the edge
Infrastructure
The latest on the world’s leading enterprise Linux platform
Applications
Inside our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Customer support
- Developer resources
- Find a partner
- Red Hat Ecosystem Catalog
- Red Hat value calculator
- Documentation
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit