Subscribe to the feed

Recently, I stumbled upon a StackOverflow question around StatefulSets which made me wonder how well understood they are at large. So I decided to put together a simple stateful app that can be used to experiment with a StatefulSet. In this blog post we will have a closer look at this app and see it in action.

If you're not familiar with StatefulSets, now is a good time for a refresher, consulting the official docs concerning their usage and guarantees they provide.

Meet mehdb, the Kubernetes-native key-value store

The stateful app I wrote is called mehdb and you can think of it as a naive distributed key-value store, supporting permanent read and write operations. In mehdb there is one leader and one or more followers: a leader accepts both read and write operations and a follower only serves read operations. If you attempt a write operation on a follower, it will redirect the request to the leader.

Both the leader and the followers are themselves stateless, the state is entirely kept and managed through the StatefulSet which is responsible for launching the pods in order and the persistent volumes, guaranteeing data being available across pod or node re-starts. Each follower periodically queries the leader for new data and syncs it.

Three exemplary interactions are shown in the architecture diagram above:

  1. Shows a WRITE operation using the /set/$KEY endpoint which is directly handled by the leader shard.
  2. Shows a READ operation using the /get/$KEY endpoint which is directly handled by a follower shard.
  3. Shows a WRITE operation issued against a follower shard using the /set/$KEY endpoint and which is redirected, using the HTTP status code 307, to the leader shard and handled there.

To ensure that followers have time to sync data before they serve reads readiness probes are used: the /status?level=full endpoint returns a HTTP 200 status code and the number of keys it can serve or a 500 otherwise.

StatefulSet In Action

In order to try out the following, you'll need a Kubernetes 1.9 (or higher) cluster. Also, the default setup defined in app.yaml assumes that a storage class ebs is defined.

Let's deploy mehdb first. The following brings up the StatefulSet including two pods (a leader and a follower), binds the persistent volumes to each pod as well as creates a headless service for it:

$ kubectl create ns mehdb
$ kubectl -n=mehdb apply -f app.yaml

First, let's verify that StatefulSet has created the leader (mehdb-0) and follower pod (mehdb-1) and that the persistent volumes are in place:

$ kubectl -n=mehdb get sts,po,pvc -o wide
statefulsets/mehdb 2 2 28m shard

po/mehdb-0 1/1 Running 0 28m ip-172-31-59-148.ec2.internal
po/mehdb-1 1/1 Running 0 25m ip-172-31-59-74.ec2.internal

pvc/data-mehdb-0 Bound pvc-f464d0c3-7527-11e8-8993-123713f594ec 1Gi RWO ebs 28m
pvc/data-mehdb-1 Bound pvc-6e448695-7528-11e8-8993-123713f594ec 1Gi RWO ebs 25m

When I inspect the StatefulSet in the OpenShift web console, it looks like this:

That looks all good so far, now let's check the service:

$ kubectl -n=mehdb describe svc/mehdb
Name: mehdb
Namespace: mehdb
Labels: app=mehdb
Selector: app=mehdb
Type: ClusterIP
IP: None
Port: <unset> 9876/TCP
TargetPort: 9876/TCP
Session Affinity: None
Events: <none>

As expected, the headless service itself has no cluster IP and created two endpoints for the pods mehdb-0 and mehdb-1 respectively. Also, the DNS configuration should be updated now to return A record entries for the pods, let's check that:

$ kubectl -n=mehdb run -i -t --rm dnscheck --restart=Never -- nslookup mehdb
nslookup: can't resolve '(null)': Name does not resolve

Name: mehdb
Address 1: mehdb-1.mehdb.mehdb.svc.cluster.local
Address 2: mehdb-0.mehdb.mehdb.svc.cluster.local

Great, we're all set to use mehdb now. Let's first write some data, in our case, we store test data under the key test:

$ kubectl -n=mehdb run -i -t --rm jumpod --restart=Never -- sh
If you don't see a command prompt, try pressing enter.
/ $ echo "test data" > /tmp/test
/ $ curl -sL -XPUT -T /tmp/test mehdb:9876/set/test
WRITE completed/ $

Note that the -L option in above curl command makes sure that if we happen to hit the follower shard we get redirected to the leader shard and the write goes through.

We should now be able to read the data from any shard, so let's try to get it directly from the follower shard:

$ kubectl -n=mehdb run -i -t --rm mehdbclient --restart=Never -- curl mehdb-1.mehdb:9876/get/test
test data

OK, now that we know we can write and read data, let's see what happens when we scale the StatefulSet, creating two more followers (note that this can take several minutes until the readiness probes pass):

$ kubectl -n=mehdb scale sts mehdb --replicas=4
$ kubectl -n=mehdb get sts
mehdb 4 4 43m

Looks like that worked out fine. But what happens if we simulate a failure, for example by deleting one of the pods, let's say mehdb-1?

$ kubectl -n=mehdb get po/mehdb-1 -o=wide
mehdb-1 1/1 Running 0 42m ip-172-31-59-74.ec2.internal

$ kubectl -n=mehdb delete po/mehdb-1
pod "mehdb-1" deleted

$ kubectl -n=mehdb get po/mehdb-1 -o=wide
mehdb-1 1/1 Running 0 2m ip-172-31-50-211.ec2.internal

We can see that the StatefulSet detected that mehdb-1 is gone, created a replacement for it with a new IP address (on a different node) and we can still get the data from this shard via curl mehdb-1.mehdb:9876/get/test thanks to the persistent volume.

When you're done, remember to clean up. The default behavior of the StatefulSet is to remove its pods as well (if you want to keep them around, use --cascade=false):

$ kubectl -n=mehdb delete sts/mehdb
statefulset "mehdb" deleted

When you delete the StatefulSet, it does not touch the persistent volumes nor the service, so we have to take care of that ourselves:

$ for i in 0 1 2 3; do kubectl -n=mehdb delete pvc/data-mehdb-$i; done
persistentvolumeclaim "data-mehdb-0" deleted
persistentvolumeclaim "data-mehdb-1" deleted
persistentvolumeclaim "data-mehdb-2" deleted
persistentvolumeclaim "data-mehdb-3" deleted

$ kubectl -n=mehdb delete svc/mehdb

With this we conclude the exploration of StatefulSets in action and if you're interested to learn more about the topic, make sure to check out the following resources:

I've been using OpenShift Online for the experiment and would love to hear from others using it in a different environment (just make sure you have Kubernetes 1.9 or above). If you want to extend mehdb you need a little working knowledge of Go: the source clocks in at some 230 lines of code and it also shows how awesome the primitives Kubernetes provides are, allowing us to build custom systems on top of a well-designed API.

About the author


Browse by channel

automation icon


The latest on IT automation for tech, teams, and environments

AI icon

Artificial intelligence

Updates on the platforms that free customers to run AI workloads anywhere

open hybrid cloud icon

Open hybrid cloud

Explore how we build a more flexible future with hybrid cloud

security icon


The latest on how we reduce risks across environments and technologies

edge icon

Edge computing

Updates on the platforms that simplify operations at the edge

Infrastructure icon


The latest on the world’s leading enterprise Linux platform

application development icon


Inside our solutions to the toughest application challenges

Original series icon

Original shows

Entertaining stories from the makers and leaders in enterprise tech