Secure your AKS kubernetes secret using Azure Key Vault provider and CSI driver

Introduction

In today’s world, securing sensitive data is paramount for any organization. One way to achieve this is by using Azure Key Vault, a cloud-based service that allows you to securely store and manage cryptographic keys, certificates, and secrets. However, managing secrets in a Kubernetes environment can be challenging, especially when scaling to large deployments with thousands of containers. This is where the Container Storage Interface (CSI) driver for Azure Key Vault comes in. In this article, we will explore the benefits of using the CSI driver with Azure Key Vault to manage secrets in Kubernetes, including enhanced security, simplified secrets management, better compliance, improved scalability, and enhanced automation. We will also discuss how to configure and use the CSI driver with Azure Key Vault in your Kubernetes environment, as well as some best practices for managing secrets in Kubernetes. By the end of this article, you will have a deeper understanding of how the CSI driver for Azure Key Vault can help you securely manage secrets in Kubernetes and improve the overall security and compliance of your containerized applications.

This is a quick end to end example of securing your secrets in AKS using the Azure Key Vault provider for secret store CSI driver. The example uses a managed user identity to access the secrets stored in Azure Key Vault.

Create an AKS cluster

Create a resource group for the AKS cluster:

az group create --name avaxia-dev --location eastus

Create an AKS cluster while enabling Azure Key Vault Provider for Secrets Store CSI Driver:

az aks create \
--resource-group avaxia-dev \
--name avaxia-dev \
--network-plugin azure \
--enable-managed-identity \
--enable-addons azure-keyvault-secrets-provider

If you have already a ready cluster , you have just to enable the addon using the below Azure CLI command:

az aks enable-addons --addons azure-keyvault-secrets-provider --name <AKS_CLUSTER_NAME> --resource-group <RESOURCE_GROUP_NAME>

Download the cluster credentials and configure kubectl to use them (only if you created a new cluster ):

az aks get-credentials --resource-group myAKSResourceGroup --name myAKSCluster

Check that the Secrets Store CSI Driver and the Azure Key Vault Provider are installed in the cluster:

kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver, secrets-store-provider-azure)'
NAME READY STATUS AGE
aks-secrets-store-csi-driver-5tnfn 3/3 Running 8h
aks-secrets-store-csi-driver-2b2bd 3/3 Running 8h
aks-secrets-store-csi-driver-92gbe 3/3 Running 8h
aks-secrets-store-provider-azure-2bd2b 1/1 Running 8h
aks-secrets-store-provider-azure-fz92g 1/1 Running 8h
aks-secrets-store-provider-azure-5tns4 1/1 Running 8h

When we enable the Azure Key Vault secret provider, the add-on will create a user assigned managed identity in the node managed resource group. Store its resource ID in a variable for later use:

KV_IDENTITY_RESOURCE_ID=$(az aks show -g avaxia-dev -n avaxia-dev --query addonProfiles.azureKeyvaultSecretsProvider.identity.clientId -o tsv)

Create Azure Key Vault

Create a resource group for Azure Key vaultaz group create –name avaxia-dev –location eastus

Create a key vault while storing its name in a variable:

KEY_VAULT_NAME=CSIDEMO
az keyvault create --name $KEY_VAULT_NAME --resource-group avaxia-dev --location eastus

Create a secret and a key in the Vault for later demonstration:

az keyvault secret set --vault-name $KEY_VAULT_NAME -n ExampleSecret --value MyAKSExampleSecret

or/and

az keyvault key create --vault-name $KEY_VAULT_NAME -n ExampleKey --protection software

Grant the AKS key vault managed identity permissions to read your key vault and view its contents:

# set policy to access keys in your key vault<br>az keyvault set-policy -n $KEY_VAULT_NAME --key-permissions get --spn $KV_IDENTITY_RESOURCE_ID

# set policy to access secrets in your key vault<br>az keyvault set-policy -n $KEY_VAULT_NAME --secret-permissions get --spn $KV_IDENTITY_RESOURCE_ID

# set policy to access certs in your key vault<br>az keyvault set-policy -n $KEY_VAULT_NAME --certificate-permissions get --spn $KV_IDENTITY_RESOURCE_ID

Create Kubernetes resources

Store the tenant ID in a variable, you can get the value from the Azure AD tenant overview page:

TENANT_ID=${{Add your tenant ID here}}

Create a SecretProviderClass by using the following YAML, using your own values for userAssignedIdentityID, keyvaultName, tenantId, and the objects to retrieve from your key vault:

cat <<EOF | kubectl apply -f -
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname-user-msi
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true" # true since using managed identity
    userAssignedIdentityID: $KV_IDENTITY_RESOURCE_ID
    keyvaultName: $KEY_VAULT_NAME
    cloudName: ""
    objects:  |
      array:
        - |
          objectName: ExampleSecret
          objectType: secret    # object types: secret, key, or cert
          objectVersion: ""     # default to latest if empty
        - |
          objectName: ExampleKey
          objectType: key
          objectVersion: ""
    tenantId: $TENANT_ID
EOF

Create a pod that mounts the secret and the key using the secret provider class we just created:

cat <<EOF | kubectl apply -f -
---
kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline-user-msi
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname-user-msi"
EOF

Validate secrets were mounted:

kubectl exec busybox-secrets-store-inline-user-msi -- ls /mnt/secrets-store/

Read the content of the secret and key:

kubectl exec busybox-secrets-store-inline-user-msi -- cat /mnt/secrets-store/ExampleSecret
kubectl exec busybox-secrets-store-inline-user-msi -- cat /mnt/secrets-store/ExampleK

The pod require to restart every time the key vault secret changes in order to pull the new secret so to avoid this manual operation , we need to activate the autorotation in AKS cluster.

az aks update --enable-secret-rotation -n CLUSTER-NAME -g RESOURCE-GROUP

Once set up, every time developers update the keyvault secret , the secret class provider will pull the new secret and inject it in the pod.Please note that there is a pulling interval time which is 2 minutes.

Don’t forget to change your code and switch it to read secret from volume path.

Conclusion

The CSI (Container Storage Interface) driver for Azure Key Vault allows you to securely retrieve secrets from Azure Key Vault and use them as Kubernetes secrets in your containerized applications.

Overall, using the CSI driver with Azure Key Vault provides a powerful and flexible method for securely managing secrets in Kubernetes, simplifying the development process and improving the security and compliance of your containerized applications.