NOTE: v2 of this walkthrough minimizes the use of jsctl and explicitly installs js-operator:v0.0.1-alpha.24 (via Helm) which has built in support for the latest version of VenafiEnhancedIssuer/VenafiConnection CRDs.
Terminology:
- TLSPK: TLS Protect for Kubernetes (previously Jetstack Secure or JSS)
- TLSPC: TLS Protect Cloud (previously Venafi as a Service or VaaS)
- TLSP: TLS Protect Data Centre (previously Venafi Trust Protection Platform or TPP)
- VEI: Venafi Enhanced Issuer (not to be confused with the native cert-manager issuer for Venafi)
cert-manager's native Venafi issuer requires Kubernetes secrets to hold Venafi credentials (e.g. API keys). Ideally you wish to eliminate the use of all secrets as these create a potential attack vector.
With the help of TLSPK js-operator, your goal is to use Venafi credentials stored off-cluster to request certificates from either TLSP or TLSPC.
In this is example your certificate issuance platform will be TLSPC and your credentials will be stored in Hashicorp Vault.
It is generally good practice to monitor your Kubernetes objects, ensuring your dependencies are in place (i.e. Running) at the point of need
watch "kubectl get pods,crds -A | grep -v system"
k8s_name=k8s-$(date +"%y%m%d%H%M")
cat <<EOF | kind create cluster --config -
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ${k8s_name}
nodes:
- role: control-plane
EOF
We aim to minimize the use of jsctl (WIP).
At this point we need it ONLY to download the private image pull secrets, as shown below.
# rm -rf ~/.jsctl/ # (optional)
jsctl auth login
jsctl config set organization gallant-wright # (change org as appropriate)
jsctl registry auth init
Using Helm (instead of jsctl) means we can install whichever version we want - in this case v0.0.1-alpha.24
kubectl create namespace jetstack-secure
dockerconfigjson=$(mktemp)
cat <<EOF > ${dockerconfigjson}
{"auths":
{"eu.gcr.io":
{"username": "_json_key",
"email": "[email protected]",
"auth": "$(echo "_json_key:$(cat ~/.jsctl/eu.gcr.io--jetstack-secure-enterprise.json)" | base64)"
}
}
}
EOF
kubectl -n jetstack-secure create secret docker-registry jse-gcr-creds \
--from-file .dockerconfigjson=${dockerconfigjson}
helm -n jetstack-secure upgrade -i js-operator \
oci://eu.gcr.io/jetstack-secure-enterprise/charts/js-operator \
--version v0.0.1-alpha.24 \
--registry-config ${dockerconfigjson} \
--set images.secret.enabled=true \
--set images.secret.name=jse-gcr-creds \
--wait
Use a single installation manifest to get the basics (including the cert-manager and approver-policy components) up and running.
kubectl create -f - << EOF
apiVersion: operator.jetstack.io/v1alpha1
kind: Installation
metadata:
creationTimestamp: null
name: jetstack-secure
spec:
approverPolicy: {}
certManager:
controller:
replicas: 1
webhook:
replicas: 1
images:
secret: jse-gcr-creds
EOF
sleep 5 && kubectl -n jetstack-secure wait --for=condition=Available=True --all deployments --timeout=-1s
This includes the Venafi(Cluster)Issuer and VenafiConnection CRDs
kubectl patch installation jetstack-secure --type merge --patch-file <(cat <<EOF
spec:
venafiEnhancedIssuer:
replicas: 1
EOF
)
sleep 5 && kubectl -n jetstack-secure wait --for=condition=Available=True --all deployments --timeout=-1s
At this point you may choose to update your watch command to monitor new CRD type objects, for example
watch "kubectl get pods,venaficonnections,venaficlusterissuers,certificaterequestpolicies -A | grep -v system"
The VenafiConnection object you are about to create will use this service account to establish communication with Vault.
Observe how this references serviceaccounts/token.
TODO would like to factor out this step somehow. Maybe if the venafi-connection service account was adjusted to support serviceaccounts/token resources, then Vault and the VenafiConnection object could both reference that existing service account instead?
cat <<EOF | kubectl -n jetstack-secure apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: tlspc-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: create-tokens-for-tlspc-issuer
rules:
- apiGroups: [""]
resources: ["serviceaccounts/token"]
verbs: ["create"]
resourceNames: ["tlspc-sa"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: tlspc-issuer-sa-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: create-tokens-for-tlspc-issuer
subjects:
- kind: ServiceAccount
name: venafi-connection
namespace: jetstack-secure
EOF
helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update
helm -n vault upgrade -i vault \
hashicorp/vault \
--create-namespace \
--set "server.dev.enabled=true" \
--wait
sleep 5 && kubectl -n vault wait -l statefulset.kubernetes.io/pod-name=vault-0 --for=condition=ready pod --timeout=-1s
Note this references tlspc-sa created previously.
kubectl exec -n vault pods/vault-0 -- \
vault auth enable -path=tlspc kubernetes
kubectl exec -n vault pods/vault-0 -- \
vault write auth/tlspc/config kubernetes_host=https://kubernetes.default.svc
kubectl exec -n vault pods/vault-0 -- \
vault write auth/tlspc/role/tlspc-role \
role_type=jwt \
bound_audiences=vault.vault.svc.cluster.local \
user_claim=sub \
bound_service_account_names=tlspc-sa \
bound_service_account_namespaces=jetstack-secure \
policies=tlspc-policy-readonly \
ttl=5m
cat <<EOF | kubectl exec -i -n vault pods/vault-0 -- vault policy write tlspc-policy-readonly -
path "secret/data/tlspc-connection/creds" {
capabilities = ["read"]
}
EOF
api_key=<TLSPC_API_KEY> # <- get this from https://ui.venafi.cloud/platform-settings/user-preferences
kubectl exec -n vault pods/vault-0 -- vault kv put -mount=secret tlspc-connection/creds api-key=${api_key}
Note how this uses tlspc-sa to pull the TLSPC API key out of Vault.
kubectl patch installation jetstack-secure --type merge --patch-file <(cat <<EOF
spec:
venafiConnections:
- name: tlspc-connection
vaas:
apiKey:
- serviceAccountToken:
name: tlspc-sa
audiences: [vault.vault.svc.cluster.local]
- hashicorpVaultOAuth:
authInputType: OIDC
role: tlspc-role
authPath: /v1/auth/tlspc/login
url: http://vault.vault.svc.cluster.local:8200
- hashicorpVaultSecret:
secretPath: /v1/secret/data/tlspc-connection/creds
fields: ["api-key"]
url: http://vault.vault.svc.cluster.local:8200
EOF
)
Create the issuer object which references the VenafiConnection object you just created.
Behind the scenes, this also creates an "open" CertificateRequestPolicy object for the issuer so all CRs will be approved by default.
Observe how this depends upon the Default\Default zone in TLSPC. Please adjust to suit.
kubectl patch installation jetstack-secure --type merge --patch-file <(cat <<EOF
spec:
issuers:
- clusterScope: true
name: tlspc-issuer
venafiEnhancedIssuer:
venafiConnectionName: tlspc-connection
zone: Default\Default
EOF
)
We can test the above works by requesting a certificate.
kubectl create namespace tests
cat << EOF | kubectl -n tests apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: www.vei-demo.com
spec:
secretName: www-vei-demo-com-tls
commonName: www.vei-demo.com
dnsNames:
- www.vei-demo.com
issuerRef:
name: tlspc-issuer
kind: VenafiClusterIssuer
group: jetstack.io
EOF
Check the CertificateRequest is "Ready".
kubectl -n tests get cr
If it is, inspect the secret.
kubectl -n tests get secret www-vei-demo-com-tls -o 'go-template={{index .data "tls.crt" | base64decode}}' | openssl x509 -noout -text | head -11
kind delete cluster --name ${k8s_name}
If you wish to connect your cluster to TLSPK (optional), you may do so as follows
k8s_name_tlspk=$(tr "-" "_" <<< ${k8s_name})
jsctl clusters connect ${k8s_name_tlspk}
# jsctl clusters delete ${k8s_name_tlspk} --force