As announced in a previous post, we will, in this and the following posts, implement a bitcoin controller for Kubernetes. This controller will be aimed at starting and operating a bitcoin test network and is not designed for production use.
Here are some key points of the design:
- A bitcoin network will be specified by using a custom resource
- This definition will contain the number of bitcoin nodes that the controller will bring up. The controller will also talk to the individual bitcoin daemons using the Bitcon JSON RPC API to make the nodes known to each other
- The controller will monitor the state of the network and maintain a node list which is part of the status subresource of the CRD
- The bitcoin nodes are treated as stateful pods (i.e. controlled by a stateful set), but we will use ephemeral storage for the sake of simplicity
- The individual nodes are not exposed to the outside world, and users running tests against the cluster either have to use tunnels or log into the pod to run tests there – this is of course something that could be changed in a future version
The primary goal of writing this operator was not to actually run it in real uses cases, but to demonstrate how Kubernetes controllers work under the hood… Along the way, we will learn a bit about building a bitcoin RPC client in Go, setting up and using service accounts with Kubernetes, managing secrets, using and publishing events and a few other things from the Kubernetes / Go universe.
Step 1: build the bitcoin Docker image
Our controller will need a Docker image that contains the actual bitcoin daemon. At least initially, we will use the image from one of my previous posts that I have published on the Docker Hub. If you decide to use this image, you can skip this section. If, however, you have your own Docker Hub account and want to build the image yourself, here is what you need to do.
Of course, you will first need to log into Docker Hub and create a new public repository.
You will also need to make sure that you have a local version of Docker up and running. Then follow the instructions below, replacing christianb93 in all but the first lines with your Docker Hub username. This will
- Clone my repository containing the Dockerfile
- Trigger the build and store the resulting image locally, using the tag username/bitcoind:latest – be patient, the build can take some time
- Log in to the Docker hub which will store your credentials locally for later use by the docker command
- Push the tagged image to the Docker Hub
- Delete your credentials again
$ git clone https://github.com/christianb93/bitcoin.git $ cd bitcoin/docker $ docker build --rm -f Dockerfile -t christianb93/bitcoind:latest . $ docker login $ docker push christianb93/bitcoind:latest $ docker logout
Step 2: setting up the skeleton – logging and authentication
We are now ready to create a skeleton for our controller that is able to start up inside a Kubernetes cluster and (for debugging purposes) locally. First, let us discuss how we package our code in a container and run it for testing purposes in our cluster.
The first thing that we need to define is our directory layout. Following standard conventions, we will place our code in the local workspace, i.e. the $GOPATH directory, under $GOPATH/src/github.com/christianb93/bitcoin-controller. This directory will contain the following subdirectories.
- internal will contain our packages as they are not meant to be used outside of our project
- cmd/controller will contain the main routine for the controller
- build will contain the scripts and Dockerfiles to build everything
- deployments will holds all manifest files needed for the deployment
By default, Go images are statically linked against all Go specific libraries. This implies that you can run a Go image in a very minimal container that contains only C runtime libraries. But we can go even further and ask the Go compiler to also statically link the C runtime library into the Go executable. This executable is then independent of any other libraries and can therefore run in a “scratch” container, i.e. an empty container. To compile our controller accordingly, we can use the commands
CGO_ENABLED=0 go build docker build --rm -f ../../build/controller/Dockerfile -t christianb93/bitcoin-controller:latest .
in the directory cmd/controller. This will build the controller and a docker image based on the empty scratch image. The Dockerfile is actually very simple:
FROM scratch # # Copy the controller binary from the context into our # container image # COPY controller / # # Start controller # ENTRYPOINT ["/controller"]
Let us now see how we can run our controller inside a test cluster. I use minikube to run tests locally. The easiest way to run own images in minikube is to build them against the docker instance running within minikube. To do this, execute the command
eval $(minikube docker-env)
This will set some environment variables so that any future docker commands are directed to the docker engine built into minikube. If we now build the image as above, this will create a docker image in the local repository. We can run our image from there using
kubectl run bitcoin-controller --image=christianb93/bitcoin-controller --image-pull-policy=Never --restart=Never
Note the image pull policy – without this option, Kubernetes would try to pull the image from the Docker hub. If you do not use minikube, you will have to extend the build process by pushing the image to a public repository like Docker hub or a local repository reachable from within the Kubernetes cluster that you use for your tests and omit the image pull policy flag in the command above. We can now inspect the log files that our controller writes using
kubectl logs bitcoin-controller
To implement logging, we use the klog package. This will write our log message to the standard output of the container, where they are picked up by the Docker daemon and forwarded to the Kubernetes logging system.
Our controller will need access to the Kubernetes API, regardless of whether we execute it locally or within a Kubernetes cluster. For that purpose, we use a command-line argument kubeconfig. If this argument is set, it refers to a kubectl config file that is used by the controller. We then follow the usual procedure to create a clientset.
In case we are running inside a cluster, we need to use a different mechanism to obtain a configuration. This mechanism is based on a service accounts.
Essentially, service accounts are “users” that are associated with a pod. When we associate a service account with a pod, Kubernetes will map the credentials that authenticate this service account into /var/run/secrets/kubernetes.io/serviceaccount. When we use the helper function clientcmd.BuildConfigFromFlags and pass an empty string as configuration file, the Go client will fall back to in-cluster configuration and try to retrieve the credentials from that location. If we do not specify a service account for the pod, a default account is used. This is what we will do for the time being, but we will soon run into trouble with this approach and will have to define a service account, an RBAC role and a role binding to grant permissions to our controller.
Step 3: create a CRD
Next, let us create a custom resource definition that describes our bitcoin network. This definition is very simple – the only property of our network that we want to make configurable at this point in time is the number of bitcoin nodes that we want to run. We do specify a status subresource which we will later use to track the status of the network, for instance the IP addresses of its nodes. Here is our CRD.
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: bitcoin-networks.bitcoin-controller.christianb93.github.com
spec:
version: v1
group: bitcoin-controller.christianb93.github.com
scope: Namespaced
subresources:
status: {}
names:
plural: bitcoin-networks
singular: bitcoin-network
kind: BitcoinNetwork
validation:
openAPIV3Schema:
properties:
spec:
required:
- nodes
properties:
nodes:
type: int
Step 4: pushing to a public repository and running the controller
Let us now go through the complete deployment cycle once, including the push to a public repository. I assume that you have a user on Docker Hub, (for me, this is christianb93), and have set up a repository called bitcoin-controller in this account. I will also assume that you have done a docker login before running the commands below. Then, building the controller is easy – simply run the following commands, replacing the christianb93 in the last two commands with your username on Docker Hub.
cd $GOPATH/src/github.com/christianb93/bitcoin-controller/cmd/controller CGO_ENABLED=0 go build docker build --rm -f ../../build/controller/Dockerfile -t christianb93/bitcoin-controller:latest . docker push christianb93/bitcoin-controller:latest
Once the push is complete, you can run the controller using a standard manifest file as the one below.
apiVersion: v1
kind: Pod
metadata:
name: bitcoin-controller
namespace: default
spec:
containers:
- name: bitcoin-controller-ctr
image: christianb93/bitcoin-controller:latest
Note that this will only pull the image from Docker Hub if we delete the local image using
docker rmi christianb93/bitcoin-controller:latest
from the minikube Docker repository (or did not use that repository at all). You will see that pushing takes some time, this is why I prefer to work with the local registry most of the time and only push to the Docker Hub once in a while.
We now have our build system in place and a working skeleton which we can run in our cluster. This version of the code is available in my GitHub repository under the v0.1 tag. In the next post, we will start to add some meat – we will model our CRD in a Go structure and put our controller in a position to react on newly added bitcoin networks.
LeftAsExercise 
1 Comment