Kubernetes Custom Resources Grow Up in v1.10

Introduction

If you are new to Kubernetes CustomResourceDefinitions (CRDs), check out our previous blog posts on what they are and how to effectively pair them with code generation to write custom Kubernetes controllers.

CustomResourceDefinitions are used to extend the Kubernetes API and create your own object types, called custom resources. Though custom resources worked well with the Kubernetes API server, there were a few differences between how they behaved and how native objects like Deployments and Pods behaved. Kubernetes 1.10 makes Custom Resources look and act more like native API objects, helping you write more efficient custom controllers.

Kubernetes API conventions

First things first - a quick overview of how the Kubernetes API ecosystem works, and how native objects behave.

Everything in the Kubernetes API is a declarative configuration object, which means it represents the desired state of the system. For example, to specify 5 replicas, you say “I want 5 replicas,” instead of asking the system to run something 5 times. Since you declare that there should be five replicas, it is called declarative configuration.

In Kubernetes objects, you generally declare your desired values in a nested object field called spec. The status of the object at the current time is depicted by a nested object field called status. The controllers within Kubernetes continuously talk to the API in a loop and make sure that the current state of the system (status) matches the desired state of the system (spec). The status of a resource is also called the status subresource for the particular resource.

Before 1.10, custom resources API endpoints did not distinguish between spec and status fields. However with 1.10, controllers can now leverage the alpha feature which introduces this difference. Spec corresponds to the .spec JSONPath in the custom resource and status corresponds to the .status JSONPath in the custom resource.

This is helpful to split the permission (RBAC rules) for accessing a custom resource:
only the controller should be able to write the status, and only read the spec
only the users should be able to write the spec, but also read the status.
If .spec and .status are split via a subresource, we declare these permissions and hence create a secure setup.

To use this feature, enable the CustomResourcesSubresources feature gate on the kube-apiserver:

--feature-gates=CustomResourceSubresources=true

Status

Here is a Database CRD from our API server deep dive series. To enable the status subresource for instances of this CRD, create a subresources stanza and declare the initial status as an empty struct:

$ cat databases-crd.yaml

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

 name: databases.example.com

spec:

 group: example.com

 version: v1

 scope: Namespaced

 names:

   plural: databases

   kind: Database

 # subresources describes the subresources for custom resources.

 subresources:

   # status enables the status subresource.

   status: {}

Then we can create the CRD from the YAML:

$ kubectl create -f databases-crd.yaml

customresourcedefinition.apiextensions.k8s.io "databases.example.com" created

Now that the CRD has been created, let’s create an instance of the CRD:

$ cat mysql-database.yaml

apiVersion: example.com/v1

kind: Database

metadata:

 name: mysql

spec:

 user: my-user

 password: secret

 replicas: 1

 encoding: unicode

Note that the spec in this YAML denotes the desired state of the database resource mysql.

$ kubectl create -f mysql-database.yaml

database.example.com "mysql" created

Using status via curl

Once you create this database instance, an endpoint depicting the status or the current state of the resource mysql is created. You can curl this endpoint to check the status. Since this endpoint depicts the status, it is called the status subresource.

/apis/example.com/v1/namespaces/default/databases/mysql/status

Using status in a controller

The status of a custom resource (.status JSONPath) depicts the current state of whatever is being managed by the controller. The status part of a resource is “privileged” in the sense that only a controller is able to update it. This is necessary to make sure that updates to the main resource do not change the status to an inconsistent state.

https://github.com/nikhita/custom-database-controller shows a controller written in Go which uses the number of replicas in the Database custom resource to scale mysql deployments. You can use the generated client with the UpdateStatus method. This will ensure that nothing other than the resource status is updated.

_, err := c.exampleclientset.ExampleV1().Databases(database.Namespace).UpdateStatus(database)

Using the status subresource also means that .metadata.generation is incremented each time the spec changes. This is very useful for controllers that support status.ObservedGeneration since they can be optimized by avoiding sync whenever the controller object is updated, but the spec hasn't changed (ObservedGeneration = Generation). Without this, you might trigger a sync as a result of updating your own status.

Scale

Like Deployments and ReplicaSets, custom resources can now be scaled. This means that custom resources now support the scale subresource along with the status subresource. The scale subresource exposes the autoscaling/v1.Scale object.

To enable the scale subresource, mention the corresponding JSONPaths in the CRD.

$ cat databases-crd.yaml

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

 name: databases.example.com

spec:

 group: example.com

 version: v1

 scope: Namespaced

 names:

   plural: databases

   kind: Database

 subresources:

   status: {}

   # scale enables the scale subresource.

   scale:

     # specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.

     specReplicasPath: .spec.replicas

     # statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.

     statusReplicasPath: .status.replicas

     # labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.

     labelSelectorPath: .status.labelSelector

When you create the CRD and the CRD instance (as above):

$ kubectl create -f databases-crd.yaml

customresourcedefinition.apiextensions.k8s.io "databases.example.com" created

$ kubectl create -f mysql-database.yaml

database.example.com "mysql" created

… a new scale subresource is exposed for the mysql database resource:

/apis/example.com/v1/namespaces/default/databases/mysql/scale

Using scale via curl

You can curl at the scale subresource endpoint to get the Scale object corresponding to the custom resource.

$ curl localhost:8080/apis/example.com/v1/namespaces/default/databases/mysql/scale

{

  "kind": "Scale",

  "apiVersion": "autoscaling/v1",

  "metadata": {

    "name": "mysql",

    "namespace": "default",

    "selfLink": "/apis/example.com/v1/namespaces/default/databases/mysql/scale",

    "uid": "b997bef6-3997-11e8-b92b-54e1ad6c2d05",

    "resourceVersion": "351",

    "creationTimestamp": "2018-04-06T12:40:45Z"

  },

  "spec": {

    "replicas": 1

  },

  "status": {

    "replicas": 0

  }

}

Using scale via kubectl

Consider the example of a custom database controller. When a database resource called mysql is created, it creates a deployment with replicas = 1. When we scale the database resource, the deployment is also scaled by the controller.

# Before scaling

$ kubectl get deployments

NAME    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE

my-user 1         1         1            1           4s
# Scaling
 $ kubectl scale --replicas=3 databases/mysql
 database.example.com "mysql" scaled
# After scaling
 $ kubectl get deployments
 NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
 my-user 3 3 3 3 1m
 

Using scale in a controller

Similar to how an UpdateStatus() method exists for the status subresource, we can generate the GetScale() and UpdateScale() methods for the scale subresource by adding the following tags on the Database type:

// +genclient

// +genclient:method=GetScale,verb=get,subresource=scale,result=k8s.io/api/autoscaling/v1.Scale

// +genclient:method=UpdateScale,verb=update,subresource=scale,input=k8s.io/api/autoscaling/v1.Scale,result=k8s.io/api/autoscaling/v1.Scale

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// Database describes a database.
 type Database struct {
 metav1.TypeMeta `json:",inline"`
 metav1.ObjectMeta `json:"metadata,omitempty"`
 Status DatabaseStatus `json:"status"`
 Spec DatabaseSpec `json:"spec"`
 }
 

The generated methods can be used as:

scale, err := c.exampleclientset.ExampleV1().Databases(database.Namespace).GetScale(database.Name, &metav1.GetOptions{})
updatedScale, err := c.exampleclientset.ExampleV1().Databases(database.Namespace).UpdateScale(database.Name, oldScale)
 

Categories

Kubernetes 1.10 introduces an interesting way to organize custom resources: Categories. With one or more Categories specified for a CRD, kubectl get can list all of the custom resources in the category. For example, in the manifest below, the databases.example.com CRD contains a storage category.

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

 name: databases.example.com

spec:

 group: example.com

 version: v1

 scope: Namespaced

 names:

   plural: databases

   singular: database

   kind: Database

   # categories is a list of grouped resources the custom resource belongs to.

   categories:

   - storage

Now, kubectl get storage can list all instances of CRDs that specify storage as a category. This is a convenient way to group CRDs by application or function.

$ kubectl create -f databases-crd.yaml

customresourcedefinition.apiextensions.k8s.io "databases.example.com" created
$ kubectl create -f mysql-database.yaml
 database.example.com "mysql" created
$ kubectl get storage
 NAME AGE
 mysql 3s
 

Kubernetes 1.10 adds significant features for CRDs and brings them closer to feature parity with native Kubernetes objects, making it easier than ever to write powerful and efficient custom controllers and other API extensions. Plans for the next few Kubernetes releases include more exciting features to make CRDs even more fun to use. Stay tuned!