Kubernetes storage under the hood part III – storage classes and provisioning

In the last post, we have seen the magic of persistent volume claims in action. In this post, we will look in more details at how Kubernetes actually manages storage.

Storage classes and provisioners

First, we need to understand the concept of a storage class. In a typical environment, there are many different types of storage. There could be some block storage like EBS or a GCE persistent disk, or some local storage like an SSD or a HDD, NFS based storage or some distributed storage like Ceph or StorageOS. So far, we have kept our persistent volume claims platform agnostic, on the other hand, there might be a need to specify in more detail what type of storage we want. This is done using a storage class.

To start, let us use kubectl to list the available storage classes in our standard EKS cluster (you will get different results if you use a different provider).

$ kubectl get storageclass
NAME            PROVISIONER             AGE
gp2 (default)   kubernetes.io/aws-ebs   3h

In this case, there is only one storage class called gp2. This is marked as the default storage class, meaning that in case we define a PVC which does not explicitly refer to a storage class, this class is chosen. Using kubectl with one of the flags --output json or --output yaml, we can get more information on the gp2 storage class. We find that there is an annotation storageclass.kubernetes.io/is-default-class which defines whether this storage class is the default storage class. In addition, there is a field provisioner which in this case is kubernetes.io/aws-ebs.

This looks like a Kubernetes provided component, so let us try to locate its source code in the Kubernetes GitHub repository. A quick search in the source tree will show you that there is a fact a manifest file defining the storage class gp2. In addition, the source tree contains a plugin which will communicate with the AWS cloud provider to manage EBS block storage.

The inner workings of this are nicely explained here. Basically, the PVC controller will use the storage class in the PVC to find the provisioner that is supposed to be used for this storage class. If a provisioner is found, it is asked to create the requested storage dynamically. If no provisioner is found, the controller will just wait until storage becomes available. An external provisioner can periodically scan unmatched volume claims and provision storage for them. It then creates a corresponding persistent storage object using the Kubernetes API so that the PVC controller can detect this storage and bind it to the claim. If you are interested in the details, you might want to take a look at the source code of the external provisioner controller and the example of the Digital Ocean provisioner using it.

So at the end of the day, the workflow is as follows.

• A user creates a PVC
• If no storage class is provided in the PVC, the default storage class is merged into the PVC
• Based on the storage class, the provisioner responsible for creating the storage is identified
• The provisioner creates the storage and a corresponding Kubernetes PV object
• The PVC is bound to the PV and available for use in Pods

Let us see how we can create our own storage classes. We have used the AWS EBS provisioner to create GP2 storage, but it does in fact support all storage types (gp2, io1, st1, sc1) offered by Amazon. Let us create a storage class which we can use to dynamically provision HDD storage of type st1.

$ kubectl apply -f - << EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: st1
provisioner: kubernetes.io/aws-ebs
parameters:
  type: st1
EOF
storageclass.storage.k8s.io/st1 created
$ kubectl get storageclass
NAME            PROVISIONER             AGE
gp2 (default)   kubernetes.io/aws-ebs   17m
st1             kubernetes.io/aws-ebs   15s

When you compare this to the default class, you will find that we have dropped the annotation which designates this class as default – there can of course only be one default class per cluster. We have again used the aws-ebs provisioner, but changed the type field to st1. Let us now create a persistent storage claim using this class.

$ kubectl apply -f - << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hdd-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: st1
  resources:
    requests:
      storage: 500Gi
EOF

If you submit this, wait until the PV has been created and then use aws ec2 describe-volumes, you will find a new AWS EBS volume of type st1 with a capacity of 500 Gi. As always, make sure to delete the PVC again to avoid unnecessary charges for that volume.

Manually provisioning volumes

So far, we have used a provisioner to automatically provision storage based on storage claims. We can, however, also provision storage manually and bind claims to it. This is done as usual – using a manifest file which describes a resource of kind PersistentVolume. To be able to link this newly created volume to a PVC, we first need to define a new storage class.

$ kubectl apply -f - << EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cold
provisioner: kubernetes.io/aws-ebs
parameters:
  type: sc1
EOF
storageclass.storage.k8s.io/cold created

Once we have this storage class, we can now manually create a persistent volume with this storage class. The following commands create a new volume with type sc1, retrieve its volume id and create a Kubernetes persistent volume linked to that EBS volume.

$ volumeId=$(aws ec2 create-volume --availability-zone=eu-central-1a --size=1024 --volume-type=sc1  --query 'VolumeId')
$ kubectl apply -f - << EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: cold-pv
spec:
  capacity:
    storage: 1024Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: cold
  awsElasticBlockStore:
    volumeID: $volumeId
EOF

Initially, this volume will be available, as there is no matching claim. Let us now create a PVC with a capacity of 512 Gi and a matching storage class.

$ kubectl apply -f - << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cold-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: cold
  resources:
    requests:
      storage: 512Gi
EOF

This will create a new PVC and bind it to the existing persistent volume. Note that the PVC will consume the entire 1 Ti volume, even though we have only requested half of that. Thus manually pre-provisioning volumes can be rather inefficient if the administrator and the teams do not agree on standard sizes for persistent volumes.

If we delete the PVC again, we see that persistent volume is not automatically deleted, as it was the case for dynamic provisioning, but survices and can be consumed again by a new matching claim. Similarly, deleting the PV will not automatically delete the underlying EBS volume, and we need to clean up manually.

Using Python to create volume claims

As usual, we close this post with some remarks on how to use Python to create and use persistent volume claims. Again, we omit the typical boilerplate code an refer to the full source code on GitHub for all the details.

First, let us look at how to create and populate a Python object that corresponds to a PVC. We need to instantiate an instance of V1PersistentVolumeClaim. This object again has the typical metadata, contains the access mode and a resource requirement.

pvc=client.V1PersistentVolumeClaim()
pvc.api_version="v1"
pvc.metadata=client.V1ObjectMeta(
                  name="my-pvc")
spec=client.V1PersistentVolumeClaimSpec()
spec.access_modes=["ReadWriteOnce"]
spec.resources=client.V1ResourceRequirements(
                  requests={"storage" : "32Gi"})
pvc.spec=spec

Once we have this, we can submit an API request to Kubernetes to actually create the PVC. This is done using the method create_namespaced_persistent_volume_claim of the API object. We can then create a pod specification that uses this PVC, for instance as part of a deployment. In this template, we first need a container specification that contains the mount point.

container = client.V1Container(
                name="alpine-ctr",
                image="httpd:alpine",
                volume_mounts=[client.V1VolumeMount(
                    mount_path="/test",
                    name="my-volume")])

In the actual Pod specification, we also need to populate the field volumes. This field contains a list of volumes, which again refer to a volume source. So we end up with the following code.

pvcSource=client.V1PersistentVolumeClaimVolumeSource(
              claim_name="my-pvc")
podVolume=client.V1Volume(
              name="my-volume",
              persistent_volume_claim=pvcSource)
podSpec=client.V1PodSpec(containers=[container], 
                         volumes=[podVolume])

Here the claim name links the volume to our previously defined PVC, and the volume name links the volume to the mount point within the container. Starting from here, the creation of a Pod using this specification is then straightforward.

This completes our short series on storage in Kubernetes. In the next post on Kubernetes, we will look at a typical application of persistent storage – stateful applications.