I wanted to start looking at a few modules helping integrate AKS with the rest of Azure.
A big integration point is identity. For many reasons, we’ll want our pods to use service principal identities:
- Access an Azure service supporting AAD-integration
- Access Azure Resource Manager (ARM) API
- Authenticate to another API using Azure AD identities
In this article, we’ll look at Azure AD Pod Identity as a simple solution to deal with this.
As usual, the code is in GitHub.
To do one of the scenarios enumerated above traditionally, we would need to get a hold to a Service Principal’s client-id and secret / certificate. We can then call Azure AD authentication API and receive an access token. We can then use that access token with different API calls (see this article for an example).
The massive drawback with that approach is the secret management: identity credentials are exposed to application and we need to rotate those secrets.
We could avoid those drawbacks by using Azure Key Vault to store our secrets. We then face the authentication bootstrapping problem. We need an identity to access Azure Key Vault, where do we store the secret / certificate of that identity?
That last issue is solved by Azure Managed Identities. A system-assigned managed identity let a service, e.g. a Virtual Machine, acquire an identity. From there, applications do not need to know the credentials of that identity. They get access tokens from the VM itself. The credentials are never known, hence never leaked. They are also managed in the sense that rotation of credentials is taken care of by the platform.
AKS pod identities rely on something similar called User assigned managed identity. This allows users to create identities, bind them to pods and have pods acquire access tokens directly, without authenticating.
In both cases the acquisition of access tokens is done by using a well known endpoint, the Azure Instance Metadata Service identity endpoint: http://169.254.169.254/metadata/identity/oauth2/token. This endpoint is only accessible from apps using the managed identity.
The GitHub repository explains the concepts well.
Basically, the deployment script deploys 3 custom resource definitions, a daemon set and a replica set (deployment).
The 3 custom resource definitions are:
The first two are then used by users to configure pods while the third one is used by a custom controller (the MIC).
The daemon set is the Node Managed Identity (NMI). Requests done to the Instance Metadata Service endpoint (http://169.254.169.254/metadata/identity/oauth2/token) are rerouted to the local pod of that daemon set. It then does a request for token on behalf of the pod.
The deployment / replicaset is the Managed Identity Controller(MIC). Like other controllers in Kubernetes (e.g. Replica Set is a controller), it watches the Kubernetes API Server for changes in pod population.
We won’t replicate the tutorial from the GitHub repo. Instead, we’ll highlight details and dive deeper on some aspects.
apiVersion: "aadpodidentity.k8s.io/v1" kind: AzureIdentity metadata: name: <a-idname> spec: type: 0 ResourceID: /subscriptions/<subid>/resourcegroups/<resourcegroup>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<managedidentity-resourcename> ClientID: <clientid>
Basically, we reflect the Azure resource inside Kubernetes.
We then create an AzureIdentityBinding which binds that identity to pods, via label selector:
apiVersion: "aadpodidentity.k8s.io/v1" kind: AzureIdentityBinding metadata: name: demo1-azure-identity-binding spec: AzureIdentity: <a-idname> Selector: <label value to match>
The resource relationships are as follow:
As usual in Kubernetes, we do not point to pods directly, as they are ephemeral. We point to labels characterising the pods. In this case, we look for a well-known label with the name aadpodidbinding.
How does that work exactly?
If a fair share of magic seemed to be involved, let’s look back in slow motion.
When we create an AzureIdentity, nothing happens beside the identity being registered as a resource in etcd (Kubernetes persistent store).
When we create an AzureIdentityBinding, the Managed Identity Controller (MIC) starts to look for pods with the specified labels. For each of those pods, it configures the pod’s routing table to reroute the Instance Metadata Service endpoint to the Node Managed Identity (NMI). The MIC then monitors for new pods with configured
There is a great summary diagram on the GitHub site showing those interactions:
Now what is the access chaining that let our app have access to a Token?
The key is that AKS is running under its own service principal. That service principal is used to access Azure Resources. For instance, when we create a Kubernetes Service of type load balancer, it is that service principal that creates an Azure Load Balancer.
That service principal also is the one requesting the access token at the end. For that reason, we must configure the service principal to have the role Managed Identity Operator on the managed user we created.
By default, the AKS service principal is contributor on the managed resource group (i.e. resource group where the Azure VMs for AKS are) but not the resource group where AKS resource is.
We can look at predefined roles by typing
az role definition list -o table | less. We can also look at Managed Identity Operator specifically by typing
az role definition list --query "[?roleName == 'Managed Identity Operator']" -o jsonc. We can see that role has the following actions:
For more details on RBAC in Azure, please consult this short training.
It really is a chain of trust here.
Now let’s quickly demo what we have learn.
Let’s do the steps lined up in the tutorial online:
- We install the infrastructure
- We create a managed identity ; we name the identity vpl-id and put it in the same resource group as our AKS cluster
- We skip the reader role step
- We assigned the Managed Identity Operator role on AKS service principal on the managed user resource
- We install the user we created in AKS
- We install the identity binding in AKS
- Finally, we deploy a single pod:
kubectl apply -f https://raw.githubusercontent.com/vplauzon/aks/master/aad-pod-identity/pod.yaml
Notice the pod has a label
aadpodidbinding: little-pod-binding. This label must match the selected in the AzureIdentityBinding. For instance:
apiVersion: "aadpodidentity.k8s.io/v1" kind: AzureIdentityBinding metadata: name: vpl-id-to-little-pod spec: AzureIdentity: vpl-id Selector: little-pod-binding
We now have a pod running which should have access to the managed identity access tokens. Let’s test that:
$ kubectl exec test-id-pod -it sh / # curl http://169.254.169.254/metadata/identity/oauth2/token/?resource=https://vault.azure.net
This should return a JSON token.
We can see that an AzureAssignedIdentity was created:
kubectl get AzureAssignedIdentity NAME AGE test-id-pod-default-vpl-id 58s
We’ve looked at Azure AD Pod Identity, an AKS extension allowing us to use managed user identity with AKS pods.
We looked under the hood to understand the concepts of the extension.
- Install Azure AD Pod Identity infrastructure in AKS
- Azure resources:
- Create a user managed identity
- Make sure the AKS service principal is “Managed Identity Operator” on it
- AKS resources:
- Create an AzureIdentity matching the User Managed Identity we just created in Auzre
- Create an AzureIdentityBinding binding the user managed identity with a pod’s label
- Request tokens from within the pod