Dynamic disks in AKS

Solution  · 

Some workloads running on Azure Kubernetes Service (AKS) requires persisting state on disk.

In general, I recommend to use external PaaS services, i.e. Azure Blob Storage, Azure SQL DB, Azure Cosmos DB, etc. . Those services take care of the stateful nature of the service, manages HA, backups, geo-replication, etc. .

Persisting state on disks in AKS comes with none of PaaS benefit. It is like running IaaS: we need to manage backups, high availability & geo replication. Those aren’t trivial to manage.

In the few cases where I need to persist on a disk, I use Kubernetes Volume. This is a K8s abstraction where we mount a volume in a pod, but the volume can be defined in different ways. Azure specifics are:

Typically, I see teams using Azure Disk Dynamic to have performant storage own by a pod and Azure File Static for storage shared across many pods with less performance. Don’t run a database on Azure File, even Premium.

In this article we’ll cover disks. The online documentation is a little misleading as the example given there doesn’t scale to a multi-pod deployment. We’ll see how to do that.

As usual, the code is on GitHub.

Deployment with Persistent Volume Claim

The online documentation of Azure Disk Dynamic does a great job of explaining the different concepts at play here:

So, let’s go with an example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: azure-managed-disk
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: managed-premium
  resources:
    requests:
      storage: 25Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: disk-deploy
  labels:
    app: busy-box-with-disk
spec:
  replicas: 1
  selector:
    matchLabels:
      app: disk-pod
  template:
    metadata:
      labels:
        app: disk-pod
    spec:
      containers:
      - name:  busy
        image: vplauzon/get-started:part2-no-redis
        ports:
        - containerPort: 80
        volumeMounts:
        - name: volume
          mountPath: /buffer
      volumes:
      - name: volume
        persistentVolumeClaim:
          claimName: azure-managed-disk

This YAML file has two resources: a PersistentVolumeClaim and a Deployment. The former defines a claim for a 25Gb disk of Premium Managed Disk. The latter deploy a pod with 1 replica mounting a volume using the claim.

We can deploy that file using:

kubectl apply -f https://raw.githubusercontent.com/vplauzon/aks/master/dynamic-disks/dynamic-with-deploy.yaml

If we look, in the managed resource group, we can see this create a new disk:

One more disk

Our cluster has 3 nodes. One more was added. Let’s open the disk:

Disk summary

We can see the disk is attached to an agent pool VM. Basically, the disk got attached to the VM running the pod.

We can also look at the tags:

Tags

We see they correspond to Persistent Volume Claim we defined. As for the persistent volume (PV) created from that claim, we can query for it:

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                                                STORAGECLASS      REASON   AGE
pvc-d53e41ee-1810-11e9-bea0-0a58ac1f072d   25Gi       RWO            Delete           Bound         default/azure-managed-disk                           managed-premium            18m

Now, let’s look at the pods:

$ kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
disk-deploy-6dfbb5cc-79bqx   1/1     Running   0          1m

We see that, as usual, the deployment created a pod with a unique name.

Scaling to multi-pods

We successfully attached a disk on a pod via a deployment.

Let’s scale that deployment to 4 pods:

kubectl patch deploy disk-deploy -p '{"spec":{"replicas":4}}'

We quickly see there is a problem here. First only one pod starts:

$ kubectl get pods -o wide
NAME                         READY   STATUS              RESTARTS   AGE   IP            NODE                       NOMINATED NODE
disk-deploy-6dfbb5cc-79bqx   1/1     Running             0          37m   10.244.2.11   aks-agentpool-22115152-2   <none>
disk-deploy-6dfbb5cc-pl9wm   0/1     ContainerCreating   0          23m   <none>        aks-agentpool-22115152-0   <none>
disk-deploy-6dfbb5cc-q89jb   1/1     Running             0          23m   10.244.2.12   aks-agentpool-22115152-2   <none>
disk-deploy-6dfbb5cc-rkmf5   0/1     ContainerCreating   0          23m   <none>        aks-agentpool-22115152-0   <none>

Important observation: only the pod on the same node as the original pod could start.

If we look in the portal, we do not see new disks getting created. Let’s look at a failing pod:

$ kubectl describe pod disk-deploy-6dfbb5cc-pl9wm
...
Events:
  Type     Reason              Age                From                               Message
  ----     ------              ----               ----                               -------
  Normal   Scheduled           19m                default-scheduler                  Successfully assigned default/disk-deploy-6dfbb5cc-pl9wm to aks-agentpool-22115152-0
  Warning  FailedAttachVolume  19m                attachdetach-controller            Multi-Attach error for volume "pvc-d53e41ee-1810-11e9-bea0-0a58ac1f072d" Volume is already used by pod(s) disk-deploy-6dfbb5cc-79bqx
  Warning  FailedMount         89s (x8 over 17m)  kubelet, aks-agentpool-22115152-0  Unable to mount volumes for pod "disk-deploy-6dfbb5cc-pl9wm_default(41c47e32-1816-11e9-bea0-0a58ac1f072d)": timeout expired waiting for volumes to attach or mount for pod "default"/"disk-deploy-6dfbb5cc-pl9wm". list of unmounted volumes=[volume]. list of unattached volumes=[volume default-token-gljst]

We see the volume failed to attach.

The reason for that is that Kubernetes has the same Persistent Volume (PV) used by each pod.

It isn’t possible for Azure Disk to be shared between VMs. Therefore, as long as pods reside on the same node than the original pod where the disk was attached, it works.

This Kubernetes mechanism clearly fails for scenarios where we want independent volumes.

Enter stateful sets

What we used so far was Kubernetes Deployments which leverages Kubernetes Replica Sets.

To achieve our scenario, we need Kubernetes Stateful sets. Stateful sets guarantee ordering and uniqueness of Pods. Pods identity survive the pod, i.e. a given pod, upon failure would be recreated with the same identity (sticky identity).

A stateful set also manages volumes differently. They are assigned in a deterministic fashion with a claim per pod.

Let’s create something similar as last example but using a stateful set:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: disk-stateful-set
  labels:
    app: busy-box-with-stateful-set-disk
spec:
  replicas: 5
  selector:
    matchLabels:
      app: stateful-disk-pod
  serviceName:  stateful-disk-set
  template:
    metadata:
      labels:
        app: stateful-disk-pod
    spec:
      containers:
      - name:  busy
        image: vplauzon/get-started:part2-no-redis
        ports:
        - containerPort: 80
        volumeMounts:
        - name: azure-managed-disk-stateful
          mountPath: /buffer
  volumeClaimTemplates:
  - metadata:
      name: azure-managed-disk-stateful
    spec:
      storageClassName: managed-premium
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 25Gi

Let’s deploy that stateful set:

$ kubectl apply -f https://raw.githubusercontent.com/vplauzon/aks/master/dynamic-disks/dynamic-with-stateful.yaml

Now if we look at the pods getting created:

$ kubectl get pods
NAME                         READY   STATUS              RESTARTS   AGE
disk-deploy-6dfbb5cc-79bqx   1/1     Running             0          2h
disk-deploy-6dfbb5cc-pl9wm   0/1     ContainerCreating   0          2h
disk-deploy-6dfbb5cc-q89jb   1/1     Running             0          2h
disk-deploy-6dfbb5cc-rkmf5   0/1     ContainerCreating   0          2h
disk-stateful-set-0          0/1     ContainerCreating   0          1m

We notice two things:

  1. The pods' names are deterministic: disk-stateful-set-0 is the name of the stateful set plus an index (0, 1, ...)
  2. The pods are brought online in sequence, not in parallel as with a deployment

With time we can see that Azure disks get created and bound to pods.

$ kubectl get pods
NAME                         READY   STATUS              RESTARTS   AGE
disk-deploy-6dfbb5cc-79bqx   1/1     Running             0          3h
disk-deploy-6dfbb5cc-pl9wm   0/1     ContainerCreating   0          3h
disk-deploy-6dfbb5cc-q89jb   1/1     Running             0          3h
disk-deploy-6dfbb5cc-rkmf5   0/1     ContainerCreating   0          3h
disk-stateful-set-0          1/1     Running             0          3m
disk-stateful-set-1          1/1     Running             0          2m
disk-stateful-set-2          1/1     Running             0          2m
disk-stateful-set-3          1/1     Running             0          1m
disk-stateful-set-4          1/1     Running             0          51s

Now, if we delete a pod to simulate a failure, the pod will be rescheduled with the same name and the same disk.

Limitations of Stateful sets

Kubernetes online documentation lists some limitations of stateful sets.

On top of those we would like to stretch out that a stateful set isn’t a magic bullet for stateful workloads.

It basically implements the equivalent of having multiple instances of a workload, each with its own disk. Failure of pods do not impact that picture.

It doesn’t implement any stateful smarts though. For instance, it doesn’t implement a master / slave configuration typical for databases.

Stateful workloads typically get those more advanced features by implementing a Kubernetes Operator. An operator defines custom resources with their custom controllers which can then, in turn, implement specific logic.

Summary

We’ve seen how we can use Azure Disk as persistent volumes in AKS.

For non-trivial scenarios, i.e. multi-pods, we turned to stateful sets.

5 responses

  1. shilpa – 2019-11-29 at 02:49

    Hello Team,

    we are deploying statefulset in AKS with dynamic PV on managed disk, for single mount scenario the sample given in volumclaimtemplate works without any issues, but when we need 2 mounts to be provisioned dynamically how do we use volumclaimtemplate ?

    could you please help us with a sample yaml with multiple metadata within volumeclaimtemplates.

    Regards Shilpa

  2. Vincent-Philippe Lauzon – 2019-12-02 at 14:55

    Does your YAML looks like the one in https://github.com/vplauzon/aks/blob/master/dynamic-disks/dynamic-with-stateful.yaml?

  3. Anonymous – 2019-12-03 at 22:29

    Hello Vincent,

    Thanks for the response. Currently working yaml looks like the one you shared in the link. I would like to know how can i add 2 metadata in volumeclaimtemplate.

    i was trying something like below snippet, which throws “UPGRADE FAILED: YAML parse error on WCSV9/templates/search-app-master.yaml: error converting YAML to JSON: yaml: line 136: found unexpected end of stream”

    volumeMounts:

    • name: index1 mountPath: /opt/WebSphere/Liberty/usr/servers/default/logs/container
    • name: index2 mountPath: /opt/WebSphere/Appserver/logs/container

    volumeClaimTemplates:

    • metadata: name: index1 namespace: test spec: accessModes: [ “ReadWriteOnce” ] storageClassName: “index1” resources: requests: storage: 3Gi
    • metadata: name: index2 namespace: test spec: accessModes: [ “ReadWriteOnce” ] storageClassName: “index2” resources: requests: storage: 3Gi

    Could you please help us resolve this error.

    Thanks in advance, Shilpa

  4. Vincent-Philippe Lauzon – 2019-12-04 at 08:48

    It does look legit. You dash-prefix it as it is an enumeration… Assuming the indentation is ok and just got lost in translation by the comment-section-engine…

    It might just be an indentation or similar gizmo.

  5. Anonymous – 2019-12-04 at 11:06

    I kept trying with indentations but with no luck. Would you be able to share any working sample snippet for my scenario?

    Thanks Shilpa

Leave a comment