AKS, App Gateway Ingress, and Key Vault TLS secrets

aks-agw-ingress-tls-notes.md

Rough notes for AKS App Gateway ingress tls

Rough notes on setting up Application Gateway Ingress Controller for AKS. This also takes advantage of key vault and Azure Key Vault Provider for Secrets Store CSI Driver. The Ingress resource will be configured to use a tls Kubernetes secret that is create via the csi driver. In this setup, TLS is terminated at the App Gateway. App Gateway does support end-to-end TLS if that is required.

Setup preview extensions

# add az cli preview extensions
az extension add --name aks-preview
az extension update --name aks-preview
az feature register --name EnablePodIdentityPreview --namespace Microsoft.ContainerService
az feature register --name AutoUpgradePreview --namespace Microsoft.ContainerService

Create AKS cluster with AAD Pod Identity addon

# create a resource group
az group create -n agic-demo-rg -l centralus

# create cluster
az aks create \
  -n agic-demo-aks \
  -g agic-demo-rg \
  --network-plugin azure \
  --enable-managed-identity \
  -a ingress-appgw \
  --appgw-name agic-demo-agw \
  --appgw-subnet-cidr "10.2.0.0/16"

# enable pod identity. this can be done on cluster create as well
az aks update -g agic-demo-rg -n agic-demo-aks --enable-pod-identity

# get cluster creds
az aks get-credentials -n agic-demo-aks -g agic-demo-rg

Setup Key Vault

If you have certs in a key vault already, you can use that vault instead of creating a new one.

I used secrets key vault here here instead of certificate resource because certificate I'm using incldues a ca bundle. The bundle is not saved to k8s secrets when using certificate directly.

# create keyvault
az keyvault create -n agic-demo-kv -g agic-demo-rg -l centralus

# load tls.crt and tls.key into keyvault
TLS_CRT=<load certificate from where it is stored>
TLS_KEY=<load key from where it is stored>

# set secrets in key vault
# an alternative is to upload to a key vault certificate instead of using individual secrets.
az keyvault secret set -n tls-crt --vault-name agic-demo-kv --value "${TLS_CRT}"
az keyvault secret set -n tls-key --vault-name agic-demo-kv --value "${TLS_KEY}"

Setup AAD pod identity

# create managed identity in azure for the application
az identity create --resource-group agic-demo-rg --name agic-demo-id
export IDENTITY_CLIENT_ID="$(az identity show -g agic-demo-rg -n agic-demo-id --query clientId -otsv)"
export IDENTITY_RESOURCE_ID="$(az identity show -g agic-demo-rg -n agic-demo-id --query id -otsv)"
export IDENTITY_PRINCIPAL_ID="$(az identity show -g agic-demo-rg -n agic-demo-id --query principalId -otsv)"

# grant required permissions to azure identity
NODE_GROUP=$(az aks show -g agic-demo-rg -n agic-demo-aks --query nodeResourceGroup -o tsv)
NODES_RESOURCE_ID=$(az group show -n $NODE_GROUP -o tsv --query "id")
az role assignment create --role "Virtual Machine Contributor" --assignee "$IDENTITY_PRINCIPAL_ID" --scope $NODES_RESOURCE_ID
az keyvault set-policy -n agic-demo-kv --object-id $IDENTITY_PRINCIPAL_ID --secret-permissions get list

# create an identity in the cluster and map it to azure identity on previous lines
export POD_IDENTITY_NAME="agic-demo-pod-identity"
export POD_IDENTITY_NAMESPACE="agic-demo"
az aks pod-identity add --resource-group agic-demo-rg --cluster-name agic-demo-aks --namespace ${POD_IDENTITY_NAMESPACE}  --name ${POD_IDENTITY_NAME} --identity-resource-id ${IDENTITY_RESOURCE_ID}

Setup key vault csi provider

The "secrets-store-csi-driver.syncSecret.enabled" option is required for creating the kubernetes secrets.

# install csi driver and enable feature to allow k8s secret resource creation
helm repo add csi-secrets-store-provider-azure https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/charts
helm install csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --namespace kube-system --set secrets-store-csi-driver.syncSecret.enabled=true

Deploy sample app

Original sample from https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/aspnetapp.yaml.

# create secret provider
kubectl apply -f secret-provider.yaml

# deploy sample app
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f ingress.yaml

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aspnetapp
  namespace: agic-demo
  labels:
    app: aspnetapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aspnetapp
  template:
    metadata:
      labels:
        app: aspnetapp
        aadpodidbinding: agic-demo-pod-identity
    spec:
      containers:
      - image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
        name: aspnetapp-image
        ports:
        - containerPort: 80
          protocol: TCP
        volumeMounts:
        - name: tls-csi
          mountPath: "/certs"
          readOnly: true
      volumes:
        - name: tls-csi
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "agic-demo-csi-provider"

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: aspnetapp
  namespace: agic-demo
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - <host name> # replace with your host name
    secretName: tls-csi
  rules:
  - host: <host name> # replace with your host name
    http:
      paths:
      - path: /
        backend:
          serviceName: aspnetapp
          servicePort: 80

secret-provider.yaml

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: agic-demo-csi-provider
  namespace: agic-demo
spec:
  provider: azure
  secretObjects:
  - secretName: tls-csi
    type: kubernetes.io/tls
    data:
    - objectName: tls.crt
      key: tls.crt
    - objectName: tls.key
      key: tls.key
  parameters:
    usePodIdentity: "true"
    keyvaultName: agic-demo-kv # replace with your own keyvault
    objects:  |
      array:
        - |
          objectName: tls-crt
          objectAlias: tls.crt
          objectType: secret
        - |
          objectName: tls-key
          objectAlias: tls.key
          objectType: secret
    tenantId: <TENANT_ID> # replace with tenant id

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: aspnetapp
  namespace: agic-demo
spec:
  selector:
    app: aspnetapp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80