The Kubernetes API is structured around resources. Typical resources that we have seen so far are pods, nodes, containers, ingress rules and so forth. These resources are built into Kubernetes and can be addresses using the kubectl command line tool, the API or the Go client.
However, Kubernetes is designed to be extendable – and in fact, you can add your own resources. These resources are defined by objects called custom resource definitions (CRD).
Setting up custom resource definitions
Confusingly enough, the definition of a custom resource – i.e. the CRD – itself is nothing but a resource, and as such, can be created using either the Kubernetes API directly or any client you like, for instance kubectl.
Suppose we wanted to create a new resource type called book that has two attributes – an author and a title. To distinguish this custom resource from other resources that Kubernetes already knows, we have to put our custom resource definition into a separate API group. This can be any string, but to guarantee uniqueness, it is good practice to use some sort of domain, for instance a GitHub repository name. As my GitHub user name is christianb93, I will use the API group christianb93.github.com for this example.
To understand how we can define that custom resource type using the API, we can take a look at its specification or the corresponding Go structures. We see that
- The CRD resource is part of the API group apiextensions.k8s.io and has version v1beta1, so the value of the apiVersion fields needs to be apiextensions.k8s.io/v1beta1
- The kind is, of course, CustomResourceDefinition
- There is again a metadata field, which is built up as usual. In particular, there is a name field
- A custom resource definition spec consists of a version, the API group, a field scope that determines whether our CRD instances will live in a cluster scope or in a namespace and a list of names
This translates into the following manifest file to create our CRD.
$ kubectl apply -f - << EOF
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: books.christianb93.github.com
spec:
version: v1
group: christianb93.github.com
scope: Namespaced
names:
plural: books
singular: book
kind: Book
EOF
customresourcedefinition.apiextensions.k8s.io/books.christianb93.github.com created
This will create a new type of resources, our books. We can access books similar to all other resources Kubernetes is aware of. We can, for instance, get a list of existing books using the API. To do this, open a separate terminal and run
kubectl proxy
to get access to the API endpoints. Then use curl to get a list of all books.
$ curl -s -X GET "localhost:8001/apis/christianb93.github.com/v1/books" | jq
{
"apiVersion": "christianb93.github.com/v1",
"items": [],
"kind": "BookList",
"metadata": {
"continue": "",
"resourceVersion": "7281",
"selfLink": "/apis/christianb93.github.com/v1/books"
}
}
So in fact, Kubernetes knows about books and has established an API endpoint for us. Note that the path contains “apis” and not “api” to indicate that this is an extension of the original Kubernetes API. Also note that the path contains our dedicated API group name and the version that we have specified.
At this point we have completed the definition of our custom resource “book”. Now let us try to actually create some books.
$ kubectl apply -f - << EOF apiVersion: christianb93.github.com/v1 kind: Book metadata: name: david-copperfield spec: title: David Copperfield author: Dickens EOF book.christianb93.github.com/my-book created
Nice – we have created our first book as an instance of our new CRD. We can now work with this book similar to a pod, a deployment and so forth. We can for instance display it using kubectl
$ kubectl get book david-copperfield NAME AGE david-copperfield 3m38s
or access it using curl and the API.
$ curl -s -X GET "localhost:8001/apis/christianb93.github.com/v1/namespaces/default/books/david-copperfield" | jq
{
"apiVersion": "christianb93.github.com/v1",
"kind": "Book",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"christianb93.github.com/v1\",\"kind\":\"Book\",\"metadata\":{\"annotations\":{},\"name\":\"david-copperfield\",\"namespace\":\"default\"},\"spec\":{\"author\":\"Dickens\",\"title\":\"David Copperfield\"}}\n"
},
"creationTimestamp": "2019-04-21T09:32:54Z",
"generation": 1,
"name": "david-copperfield",
"namespace": "default",
"resourceVersion": "7929",
"selfLink": "/apis/christianb93.github.com/v1/namespaces/default/books/david-copperfield",
"uid": "70fbc120-6418-11e9-9fbf-080027a84e1a"
},
"spec": {
"author": "Dickens",
"title": "David Copperfield"
}
}
Validations
If we look again at what we have done and where we have started, somethings still feels a bit wrong. Remember that we wanted to define a resource called a “book” that has a title and an author. We have used those fields when actually creating a book, but we have not referred to it at all in the CRD. How does the Kubernetes API know which fields a book actually has?
The answer is simple – it does not know this at all. In fact, we can create a book with any collection of fields we want. For instance, the following will work just fine.
$ kubectl apply -f - << EOF apiVersion: christianb93.github.com/v1 kind: Book metadata: name: moby-dick spec: foo: bar EOF book.christianb93.github.com/moby-dick created
In fact, when you run this, the Kubernetes API server will happily take your JSON input and store it in the etcd that keeps the cluster state – and it will store there whatever you provide. To avoid this, let us add a validation rule to our resource definition. This allows you to attach an OpenAPI schema to your CRD against which the books will be validated. Here is our updated CRD manifest file to make this work.
$ kubectl delete crd books.christianb93.github.com
customresourcedefinition.apiextensions.k8s.io "books.christianb93.github.com" deleted
$ kubectl apply -f - << EOF
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: books.christianb93.github.com
spec:
version: v1
group: christianb93.github.com
scope: Namespaced
subresources:
status: {}
names:
plural: books
singular: book
kind: Book
validation:
openAPIV3Schema:
properties:
spec:
required:
- author
- title
properties:
author:
type: string
title:
type: string
EOF
customresourcedefinition.apiextensions.k8s.io/books.christianb93.github.com created
If you know repeat the command above, you will find that "David Copperfield" can be created, but "Moby Dick" is rejected, as it does not match the validation rules (the required fields author and title are missing).
There is another change that we have made in this version of our CRD – we have added a subresource called status to our CRD. This subresource allows a controller to update the status of the resource indepently of the specification – see the corresponding API convention for more details on this.
The controller pattern
As we have seen above, a CRD is essentially allowing you to store data as part of the cluster state kept in the etcd key-value store using a Kubernetes API endpoint. However, CRDs do not actually trigger any change in the cluster. If you POST a custom resource like a book to the Kubernetes API server, all it will do is to store that object in the etcd store.
It might come as a bit of a surprise, but strictly speaking, this is true for built-in resources as well. Suppose, for instance, that you use kubectl to create a deployment. Then, kubectl will create a PUT request for a deployment and send it to the API server. The API server will process the request and store the new deployment in the etcd. It will, however, not actually create pods, spin up containers and so forth.
This is the job of another component of the Kubernetes architecture – the controllers. Essentially, a controller is monitoring the etcd store to keep track of its contents. Whenever a new resource, for example a deployment, is created, the controller will trigger the associated actions.
Kubernetes comes with a set of built-in controllers in the controller package. Essentially, there is one controller for each type of resource. The deployment controller, for instance, monitors deployment objects. When a new deployment is created in the etcd store, it will make sure that there is a matching replica set. These sets are again managed by another controller, the replica set controller, which will in turn create matching pods. The pods are again monitored by the scheduler that determines the node on which the pods should run and writes the bindings back to the API server. The updated bindings are then picked up by the kubelet and the actual containers are started. So essentially, all involved components of the Kubernetes architecture talk to the etcd via the API server, without any direct dependencies.
Of course, the Kubernetes built-in controllers will only monitor and manage objects that come with Kubernetes. If we create custom resources and want to trigger any actual action, we need to implement our own controllers.
Suppose, for instance, we wanted to run a small network of bitcoin daemons on a Kubernetes cluster for testing purposes. Bitcoin daemons need to know each other and register themselves with other daemons in the network to be able to exchange messages. To manage that, we could define a custom resource BitcoinNetwork which contains the specification of such a network, for instance the number of nodes. We could then write a controller which
- Detects new instances of our custom resource
- Creates a corresponding deployment set to spin up the nodes
- Monitors the resulting pods and whenever a pod comes up, adds this pod to the network
- Keeps track of the status of the nodes in the status of the resource
- Makes sure that when we delete or update the network, the corresponding deployments are deleted or updated as well
Such a controller would operate by detecting newly created or changed BitcoinNetwork resources, compare their definition to the actual state, i.e. existing deployments and pods, and update their state accordingly. This pattern is known as the controller pattern or operator pattern. Operators exists for many applications, like Postgres, MySQL, Prometheus and many others.
I did actually pick this example for a reason – in an earlier post, I showed you how to set up and operate a small bitcoin test network based on Docker and Python. In the next few posts, we will learn how to write a custom controller in Go that automates all this on top of Kubernetes! To achieve this, we will first analyze the components of a typical controller – informes, queues, caches and all that – using the Kubernetes sample controller and then dig into building a custom bitcoin controller armed with this understanding.
LeftAsExercise 