vault-test.md

Minimal working example for the hot-looping issue with Vault

Following Irbe's instructions cert-manager/cert-manager#3897 (comment):

Same issue here on Slack, I was not able to reproduce this then, but seem to be able to reproduce it now with cert-manager v1.3.1.

To reproduce follow steps here except for:

1. Deploy a newer version of cert-manager and use v1 not v1alpha2 api when creating cert-manager resources
2. Set max_ttl to 720h (30d) when creating the Vault role that `cert-manager will use

vault write pki/roles/example-dot-com \
    allowed_domains=example.com \
    allow_subdomains=true \
    max_ttl=720h

After creating the Certificate cert-manager entered a continuous renewal loop, see status of the Certificate:

status:
  conditions:
  - lastTransitionTime: "2021-05-07T09:05:36Z"
    message: Certificate is up to date and has not expired
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready
  - lastTransitionTime: "2021-05-07T09:06:50Z"
    message: Renewing certificate as renewal was scheduled at 2021-05-07 09:06:50
      +0000 UTC
    observedGeneration: 1
    reason: Renewing
    status: "True"
    type: Issuing
  nextPrivateKeySecretName: example-com-l2g4c
  notAfter: "2021-06-06T09:06:50Z"
  notBefore: "2021-05-07T09:06:20Z"
  renewalTime: "2021-05-07T09:06:50Z"
  revision: 185

---

The steps I took:

curl -L https://gist.githubusercontent.com/maelvls/c23558b717422c4c648f4258a7f2fb19/raw/5a62824173522ef6030d7a27d14f0359586af91a/give-me-my-cluster > /tmp/give-me-my-cluster
install /tmp/give-me-my-cluster ~/bin


give-me-my-cluster up testcluster --cm-version v1.3.1 \
  --project jetstack-mael-valais --clouddns-zone jetstacker-mael-valais --license $(lpass show inlets-pro -p) 

 helm upgrade --install vault hashicorp/vault \
    --set server.dev.enabled=true \
    --set server.ingress.enabled=true \
    --set "server.ingress.tls[0].secretName=vault-tls" \
    --set "server.ingress.annotations.cert-manager\.io/issuer=letsencrypt" \
    --set "server.ingress.annotations.traefik\.ingress\.kubernetes\.io/redirect-entry-point=websecure" \
    --set-string "server.ingress.annotations.traefik\.ingress\.kubernetes\.io/router\.tls=true" \
    --set "server.ingress.annotations.kubernetes\.io/ingress\.class=traefik" \
    --set "server.ingress.hosts[0].host=vault.$(kubectl -n kube-system get cm domain -ojsonpath='{.data.domain}')" \
    --set "server.ingress.tls[0].hosts[0]=vault.$(kubectl -n kube-system get cm domain -ojsonpath='{.data.domain}')" \
    --set "server.ingress.tls[0].hosts[0]=vault.$(kubectl -n kube-system get cm domain -ojsonpath='{.data.domain}')" \
    --set injector.enabled=false \
    --set global.tlsDisable=true

export VAULT_ADDR="https://vault.$(kubectl -n kube-system get cm domain -ojsonpath='{.data.domain}')"
vault login -method=token token=root
vault secrets enable pki
vault secrets tune -max-lease-ttl=8760h pki
vault write pki/root/generate/internal \
    common_name=example.com \
    ttl=8760h
vault write pki/config/urls \
    issuing_certificates=$VAULT_ADDR/v1/pki/ca \
    crl_distribution_points=$VAULT_ADDR/v1/pki/crl
vault write pki/roles/example-dot-com \
    allowed_domains=example.com \
    allow_subdomains=true \
    max_ttl=720h                                    # 🔥 Irbe instruction (2)
vault policy write pki - <<EOF
path "pki*"                        { capabilities = ["read", "list"] }
path "pki/roles/example-dot-com"   { capabilities = ["create", "update"] }
path "pki/sign/example-dot-com"    { capabilities = ["create", "update"] }
path "pki/issue/example-dot-com"   { capabilities = ["create"] }
EOF
vault auth enable kubernetes
vault write auth/kubernetes/config \
    token_reviewer_jwt="$(kubectl exec vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://$(kubectl exec vault-0 -- sh -c 'echo $KUBERNETES_PORT_443_TCP_ADDR'):443" \
    kubernetes_ca_cert=@<(kubectl exec vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt)
vault write auth/kubernetes/role/issuer \
    bound_service_account_names=issuer \
    bound_service_account_namespaces=default \
    policies=pki \
    ttl=20m
kubectl create serviceaccount issuer
tee /dev/stderr <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1                        # 🔥 Irbe instruction (1)
kind: Issuer
metadata:
  name: vault-issuer
  namespace: default
spec:
  vault:
    server: http://vault.default
    path: pki/sign/example-dot-com
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: issuer
        secretRef:
          name: $(kubectl get serviceaccount issuer -o json | jq -r ".secrets[].name")
          key: token
EOF
kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1                        # 🔥 Irbe instruction (1)
kind: Certificate
metadata:
  name: example-com
  namespace: default
spec:
  secretName: example-com-tls
  issuerRef:
    name: vault-issuer
  commonName: www.example.com
  dnsNames:
  - www.example.com
EOF

# Notice the hot looping

# Now, let's upgrade from 1.3.1 to 1.4.0-beta.1:
give-me-my-cluster up testcluster --cm-version v1.4.0-beta.1 \
  --project jetstack-mael-valais --clouddns-zone jetstacker-mael-valais --license $(lpass show inlets-pro -p)

# Notice that the hot looping is gone