Azure AD Pod Identity in AKS


I wanted to start looking at a few modules helping integrate AKS with the rest of Azure.

A big integration point is identity. For many reasons, we’ll want our pods to use service principal identities:

  1. Access an Azure service supporting AAD-integration
  2. Access Azure Resource Manager (ARM) API
  3. Authenticate to another API using Azure AD identities

In this article, we’ll look at Azure AD Pod Identity as a simple solution to deal with this.

This feature currently (as of mid February 2019) is marked as in development. It is available on GitHub but should be integrated in the AKS service in a few months.

As usual, the code is in GitHub.

Traditional Approach

To do one of the scenarios enumerated above traditionally, we would need to get a hold to a Service Principal’s client-id and secret / certificate. We can then call Azure AD authentication API and receive an access token. We can then use that access token with different API calls (see this article for an example).

The massive drawback with that approach is the secret management: identity credentials are exposed to application and we need to rotate those secrets.

We could avoid those drawbacks by using Azure Key Vault to store our secrets. We then face the authentication bootstrapping problem. We need an identity to access Azure Key Vault, where do we store the secret / certificate of that identity?

Managed Identities

That last issue is solved by Azure Managed Identities. A system-assigned managed identity let a service, e.g. a Virtual Machine, acquire an identity. From there, applications do not need to know the credentials of that identity. They get access tokens from the VM itself. The credentials are never known, hence never leaked. They are also managed in the sense that rotation of credentials is taken care of by the platform.

AKS pod identities rely on something similar called User assigned managed identity. This allows users to create identities, bind them to pods and have pods acquire access tokens directly, without authenticating.

In both cases the acquisition of access tokens is done by using a well known endpoint, the Azure Instance Metadata Service identity endpoint: http://169.254.169.254/metadata/identity/oauth2/token. This endpoint is only accessible from apps using the managed identity.

Concepts

The GitHub repository explains the concepts well.

Basically, the deployment script deploys 3 custom resource definitions, a daemon set and a replica set (deployment).

The 3 custom resource definitions are:

  • AzureIdentity
  • AzureIdentityBinding
  • AzureAssignedIdentity

The first two are then used by users to configure pods while the third one is used by a custom controller (the MIC).

The daemon set is the Node Managed Identity (NMI). Requests done to the Instance Metadata Service endpoint (http://169.254.169.254/metadata/identity/oauth2/token) are rerouted to the local pod of that daemon set. It then does a request for token on behalf of the pod.

The deployment / replicaset is the Managed Identity Controller(MIC). Like other controllers in Kubernetes (e.g. Replica Set is a controller), it watches the Kubernetes API Server for changes in pod population.

Configuration

We won’t replicate the tutorial from the GitHub repo. Instead, we’ll highlight details and dive deeper on some aspects.

To configure a pod to use a pod identity, we first create a user managed identity. We then create an AzureIdentity resource, for instance:

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
 name: <a-idname>
spec:
 type: 0
 ResourceID: /subscriptions/<subid>/resourcegroups/<resourcegroup>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<managedidentity-resourcename>
 ClientID: <clientid>

Basically, we reflect the Azure resource inside Kubernetes.

We then create an AzureIdentityBinding which binds that identity to pods, via label selector:

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
 name: demo1-azure-identity-binding
spec:
 AzureIdentity: <a-idname>
 Selector: <label value to match>

That’s it.

The resource relationships are as follow:

Bindings

As usual in Kubernetes, we do not point to pods directly, as they are ephemeral. We point to labels characterising the pods. In this case, we look for a well-known label with the name aadpodidbinding.

How does that work exactly?

If a fair share of magic seemed to be involved, let’s look back in slow motion.

When we create an AzureIdentity, nothing happens beside the identity being registered as a resource in etcd (Kubernetes persistent store).

When we create an AzureIdentityBinding, the Managed Identity Controller (MIC) starts to look for pods with the specified labels. For each of those pods, it configures the pod’s routing table to reroute the Instance Metadata Service endpoint to the Node Managed Identity (NMI). The MIC then monitors for new pods with configured

There is a great summary diagram on the GitHub site showing those interactions:

Concepts

Security

Now what is the access chaining that let our app have access to a Token?

The key is that AKS is running under its own service principal. That service principal is used to access Azure Resources. For instance, when we create a Kubernetes Service of type load balancer, it is that service principal that creates an Azure Load Balancer.

That service principal also is the one requesting the access token at the end. For that reason, we must configure the service principal to have the role Managed Identity Operator on the managed user we created.

By default, the AKS service principal is contributor on the managed resource group (i.e. resource group where the Azure VMs for AKS are) but not the resource group where AKS resource is.

We can look at predefined roles by typing az role definition list -o table | less. We can also look at Managed Identity Operator specifically by typing az role definition list --query "[?roleName == 'Managed Identity Operator']" -o jsonc. We can see that role has the following actions:

  • Microsoft.ManagedIdentity/userAssignedIdentities/*/read
  • Microsoft.ManagedIdentity/userAssignedIdentities/*/assign/action
  • Microsoft.Authorization/*/read
  • Microsoft.Insights/alertRules/*
  • Microsoft.Resources/subscriptions/resourceGroups/read
  • Microsoft.Resources/deployments/*
  • Microsoft.Support/*

For more details on RBAC in Azure, please consult this short training.

It really is a chain of trust here.

Demo

Now let’s quickly demo what we have learn.

Let’s do the steps lined up in the tutorial online:

Notice the pod has a label aadpodidbinding: little-pod-binding. This label must match the selected in the AzureIdentityBinding. For instance:

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
    name: vpl-id-to-little-pod
spec:
    AzureIdentity: vpl-id
    Selector:  little-pod-binding

We now have a pod running which should have access to the managed identity access tokens. Let’s test that:

$ kubectl exec test-id-pod -it sh
/ # curl  http://169.254.169.254/metadata/identity/oauth2/token/?resource=https://vault.azure.net

This should return a JSON token.

We can see that an AzureAssignedIdentity was created:

kubectl get AzureAssignedIdentity

NAME                         AGE
test-id-pod-default-vpl-id   58s

Summary

We’ve looked at Azure AD Pod Identity, an AKS extension allowing us to use managed user identity with AKS pods.

We looked under the hood to understand the concepts of the extension.

In terms of use, Pod identities are quite simple:

  • Install Azure AD Pod Identity infrastructure in AKS
  • Azure resources:
    • Create a user managed identity
    • Make sure the AKS service principal is “Managed Identity Operator” on it
  • AKS resources:
    • Create an AzureIdentity matching the User Managed Identity we just created in Auzre
    • Create an AzureIdentityBinding binding the user managed identity with a pod’s label
  • Request tokens from within the pod
Advertisements

3 thoughts on “Azure AD Pod Identity in AKS

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s