Learning Go with Kubernetes IV – life of a request

So far we have described how a client program utilizes the Kubernetes library to talk to a Kubernetes API server. In this post, we will actually look into the Kubernetes client code and try to understand how it works under the hood.

When we work with the Kubernetes API and try to understand the client code, we first have to take a look at how the API is versioned.

Versioning in the Kubernetes API

The Kubernetes API is a versioned API. Every resource that you address using requests like GET or POST has a version. For stable versions, the version name is of the form vX, where X is an integer. In addition, resources are grouped into API groups. The full relative path to a resource is of the form

/api/GROUP/VERSION/namespaces/NAMESPACE

Let us take the job resource as an example. This resource is in the API group batch. A GET request for a job called my-job in the default namespace using version v1 of the API would therefore be something like

GET /api/batch/v1/namespaces/default/jobs/my-job

An exception is made by the core API group, which is omitted in the URL path for historical reasons. In a manifest file, API group and version are both stored in the field apiVersion, which, in our example, would be batch/v1.

Within the Kubernetes Go client, the combination of a type of resource (a “kind”, like a Pod), a version and an API group is stored in a Go structure called GroupVersionKind. In fact, this structure is declared as follows

type GroupVersionKind struct {
	Group   string
	Version string
	Kind    string
}

in the file k8s.io/apimachinery/pkg/runtime/schema/group_version.go. In the source code, instances of this class are typically called gvk. We will later see that, roughly speaking, the client contains a machinery which allows us to map forth and back between possible combinations of group, version and kind and Go structures.

An overview of the Kubernetes client

At least as far as we are concerned with getting, updating and deleting resources, the Kubernetes client code consists of the following core components:

• A clientset is the entry point into the package and typically created from a client configuration as stored in the file ~/.kube/config
• For each combination of API group and version, there is a package that contains the corresponding clients. For each resource in that group, like a Node, there is a corresponding Interface that allows us to perform operations like get, list etc. on the resource, and a corresponding object like a node itself
• The package k8s.io/client-go/rest contains the code to create, submit and process REST requests. There is a REST client, request and result structures, serializers and configuration objects
• The package k8s.io/apimachinery/pkg/runtime contains the machinery to translate API requests and replies from and to Go structures. An Encoder is able to write objects to a stream. A Decoder transforms a stream of bytes into an object. A Scheme is able to map a group-version-kind combination into a Go type and vice versa.
• The same package contains a CodecFactory that is able to create encoders and decoders, and some standard encoders and decoders, for instance for JSON and YAML

Let us now dive into each of these building blocks in more detail.

Clientsets and clients

In our first example program, we have used the following lines to connect to the API.

clientset, err := kubernetes.NewForConfig(config)
coreClient := clientset.CoreV1()
nodeList, err := coreClient.Nodes().List(metav1.ListOptions{})

Let us walk through this and see how each of these lines is implemented behind the scenes. The first line creates an instance of the class ClientSet. Essentially, a clientset is a set of client objects, where each client object represents one version of an API group. When we access nodes, we will use the API group core, and correspondingly use the field coreV1 of this structure.

This core client is an instance of k8s.io/client-go/kubernetes/typed/core/v1/coreV1Client and implementing the interface CoreV1Interface. This interface declares for each resource in the core API a dedicated getter function which returns an interface to work with this resource. For a node, the getter function is called Nodes and returns a class implementing the interface NodeInterface, which defines all the functions we are looking for – get, update, delete, list and so forth.

An instance of the Nodes class also contains a reference to a RESTClient which is the working horse where the actual REST requests to the Kubernetes API will be assembled and processed – let us continue our analysis there.

RESTClient and requests

How do REST clients work? Going back to our example code, we invoke the REST client in the line

nodeList, err := coreClient.Nodes().List(metav1.ListOptions{})

Here we invoke the List method on the nodes object which is defined in the file k8s.io/client-go/kubernetes/typed/core/v1/node.go. The core of this method is the following code snippet.

err = c.client.Get().
	Resource("nodes").
	VersionedParams(&opts, scheme.ParameterCodec).
	Timeout(timeout).
	Do().
	Into(result)

Let us go through this step by step to see what is going on. First, the attribute client referenced here is a RESTClient, defined in k8s.io/client-go/rest/client.go. Among other things, this class contains a set of methods to manipulate requests.

The first method that we call is Get, which returns an instance of the class rest.Request, defined in rest.go in the same directory. A request contains a reference to a HTTPClient, which typically is equal to the HTTPClient to which the RESTClient itself refers. The request created by the Get method will be pre-initialized with the verb “GET”.

Next, several parameters are added to the request. Each of the following methods is a method of the Request object and again returns a request, so that chaining becomes possible. First, the method Resource sets the name of the resource that we want to access, in this case “nodes”. This will become part of the URL. Then we use VersionedParams to add the options to the request and Timeout to set a timeout.

We then call Do() on the request. Here is the key section of this method

var result Result
err := r.request(func(req *http.Request, resp *http.Response) {
	result = r.transformResponse(resp, req)
})
return result

In the first line, we create an (empty) rest.Result object. In addition to the typical attributes that you would expect from a HTTP response, like a body, i.e a sequence of bytes, this object also contains a decoder, which will become important later on.

We then invoke the request method of the Request object. This function assembles a new HTTP request based on our request, invokes the Do() method on the HTTP client and then calls the provided function which is responsible for converting the HTTP response into a Result object. The default implementation of this is transformResponse, which also sets a decoder in the Result object respectively copies the decoder contained in the request object.

When all this completes, we have a Result object in our hands. This is still a generic object, we have a response body which is a stream of bytes, not a typed Go structure.

This conversion – the unmarshalling – is handled by the method Into. This method accepts as an argument an instance of the type runtime.Object and fills that object according to the response body. To understand how this work, we will have to take a look at schemes, codec factories and decoders.

Schemes and decoder

In the first section, we have seen that API resources are uniquely determined by the combination of API group, version and kind. For each valid combination, our client should contain a Go structure representing this resource, and conversely, for every valid resource in the Go world, we would expect to have a combination of group, version and kind. The translation between these two worlds is accomplished by a scheme. Among other things, a scheme implements the following methods.

• A method ObjectKind which returns all known combinations of kind, API group and version for a given object
• A method Recognizes which in turn determines whether a given combination of kind, API group and version is allowed
• A method New which is able to create a new object for a given combination of API group, kind and version

Essentially, a scheme knows all combinations of API group, version and kind and the corresponding Go structures and is able to create and default the Go structures. For this to work, all resources handled by the API need to implement the interface runtime.Object.

This is nice, but what we need to transform the result of a call to the Kubernetes API into a Go structure is a decoder object. To create decoder (and encoder) objects, the API uses the class CodecFactory. A codec factory refers to a scheme and is able to create encoders and decoders. Some of the public methods of such an object are collected in the interface NegotiatedSerializer.

This interface provides the missing link between a REST client and the scheme and decoder / encoder machinery. In fact, a REST client has an attribute contentConfig which is an object of type ContentConfig. This object contains the HTTP content type, the API group and version the client is supposed to talk to and a NegotiatedSerializer which will be used to obtain decoders and encoders.

Where are schemes and codec factories created and stored? Within the package k8s.io/client-go/kubernetes/scheme, there is a public variable Scheme and a public variable Codecs which is a CodecFactory. Both variables are declared in register.go. The scheme is initially empty, but in the init method of the package, the scheme is built up by calling (via a function list called a scheme builder) the function AddToScheme for each known API group.

Putting it all together

Armed with this understanding of the class structures, we can now again try to understand what happens when we submit our request to list all nodes.

During initialization of the package k8s.io/client-go/kubernetes/scheme, the initialization code in the file register.go is executed. This will initialize our scheme and a codec factory. As part of this, a standard decoder for the JSON format will be created (this happens in the function NewCodecFactory in codec_factory.go).

Then, we create our clientset using the function NewForConfig in the kubernetes package, which calls the method NewForConfig for each of the managed clients, including our CoreV1Client. Here, the following things happen:

• We set group and version to the static values provided in the file register.go of the v1 package – the group will be empty, as we are in the special case for the core client, and the version will be “”v1”
• We add a reference to the CodecFactory to our configuration
• We create a REST client with a base URL constructed from the host name and port of the Kubernetes API server, the API group and the version as above
• We then invoke the function createSerializers in the rest package. This function retrieves all supported media types from the codec factory and matches it against the media type in the kubectl config. Then a rest.Serializer is selected which matches group, version and media type
• The REST client is added to the core client and the core client is returned
• When we subsequently create a request using this REST client, we add this serializer to the request from which it is later copied to the result

At this point, we are ready to use this core client. We now navigate to a NodeInterface and call its list method. As explained above, this will eventually take us to the function Into defined in request.go. Here, we invoke the Decode method of our REST decoder. As this is the default JSON serializer, this method is the Decode function in json.go. This decode function first uses the scheme to determine if API group, version and kind of the response are valid and match the expected Go type. It then uses a standard JSON unmarshaller to perform the actual decoding.

This completes our short tour through the structure of the Go client source code. We have seen some central concepts of the client library – API groups, versions and kinds, schemes, encoder and decoder and the various client types – which we will need again in a later post when we discuss custom controllers.