Introduction

Istio consists of several services that intercommunicate to form the service mesh control-plane. The Istio data-plane consists of a proxy mesh, where a proxy is deployed as a sidecar with each end-user application service. Mutual TLS (mTLS) can be used to secure end-user services running within the mesh and for securing Istio control-plane communication. This document provides detailed instructions for verifying Istio mTLS certificates.

Prerequisites

Access to a v1.9 or newer Kubernetes cluster for running Istio.
A host with openssl, base64, kubectl, helm and a credential file for accessing the Kubernetes cluster where Istio will run.
Install Istio v0.8 using Helm.
Customize the installation to enable control-plane mTLS by setting the following Helm parameter:
```
--set global.controlPlaneSecurityEnabled=true
```
Customize the installation to enable data-plane mTLS by setting the following Helm parameter:
```
--set global.mtls.enabled=true
```

You can verify control-plane and data-plane mTLS is enabled by viewing the istio configmap details:

$ kubectl get cm/istio -n istio-system -o yaml | grep MUTUAL_TLS
    authPolicy: MUTUAL_TLS
      controlPlaneAuthPolicy: MUTUAL_TLS

Istio Control-Plane Verification

Providing strong identity to each service is the foundation of secure communication. Kubernetes service accounts are used to identify each Istio control-plane service. Use the kubectl get sa command to view a list of service accounts:

$ kubectl get sa -n istio-system
NAME                                     SECRETS   AGE
default                                  1         6h
istio-citadel-service-account            1         6h
istio-cleanup-old-ca-service-account     1         6h
istio-egressgateway-service-account      1         6h
istio-galley-service-account             1         6h
istio-ingress-service-account            1         6h
istio-ingressgateway-service-account     1         6h
istio-mixer-post-install-account         1         6h
istio-mixer-service-account              1         6h
istio-pilot-service-account              1         6h
istio-sidecar-injector-service-account   1         6h
prometheus                               1         6h

The default service account (sa) is used for Istio data-plane services and all other service accounts are used for Istio control-plane services. Here is an example of the service account used for Pilot:

$ kubectl get sa/istio-pilot-service-account -n istio-system -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: 2018-07-24T15:35:22Z
  labels:
    app: istio-pilot
    chart: pilot-0.8.0
    heritage: Tiller
    release: istio
  name: istio-pilot-service-account
  namespace: istio-system
  resourceVersion: "105993"
  selfLink: /api/v1/namespaces/istio-system/serviceaccounts/istio-pilot-service-account
  uid: 2dae3f35-8f57-11e8-aaf2-00505693a589
secrets:
- name: istio-pilot-service-account-token-s8ndg

Each sa contains a Kubernetes secret which holds the CA certificate and token used for submitting a Certificate Signing Request (CSR) to Citadel, the Istio Certificate Authority (CA) server. Here is an example of the service account token for Pilot:

$ kubectl get secret/istio-pilot-service-account-token-s8ndg -n istio-system -o yaml
apiVersion: v1
data:
  ca.crt: REDACTED
  namespace: aXN0aW8tc3lzdGVt
  token: REDACTED
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: istio-pilot-service-account
    kubernetes.io/service-account.uid: 2dae3f35-8f57-11e8-aaf2-00505693a589
  creationTimestamp: 2018-07-24T15:35:22Z
  name: istio-pilot-service-account-token-s8ndg
  namespace: istio-system
  resourceVersion: "105988"
  selfLink: /api/v1/namespaces/istio-system/secrets/istio-pilot-service-account-token-s8ndg
  uid: 2db0789b-8f57-11e8-aaf2-00505693a589
type: kubernetes.io/service-account-token

Citadel watches the Kubernetes API Server, creates a SPIFFE key/certificate pair for each service account, and sends them to the API Server. Use the kubectl get secret command to view the keys and certificates. Here is an example of the secret containing the certificate and key for Pilot:

$ kubectl get secret/istio.istio-pilot-service-account -n istio-system -o yaml
apiVersion: v1
data:
  cert-chain.pem: REDACTED
  key.pem: REDACTED
  root-cert.pem: REDACTED
kind: Secret
metadata:
  annotations:
    istio.io/service-account.name: istio-pilot-service-account
  creationTimestamp: 2018-07-24T15:35:31Z
  name: istio.istio-pilot-service-account
  namespace: istio-system
  resourceVersion: "106416"
  selfLink: /api/v1/namespaces/istio-system/secrets/istio.istio-pilot-service-account
  uid: 32c48097-8f57-11e8-aaf2-00505693a589
type: istio.io/key-and-cert

Each Istio control-plane pod needs to reference the secret. The data (i.e. certificates) within the secret is used by mounting a volume to each Istio control-plane pod. Here is an example of the Istio Pilot pod:

kubectl get po/istio-pilot-d5bbc5c59-w89kf -n istio-system -o yaml                  
<SNIP>
    volumeMounts:
    - mountPath: /etc/certs
      name: istio-certs
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: istio-pilot-service-account-token-s8ndg # Token and CA cert used to issue CSR
      readOnly: true
  - args:
    - proxy
    - --serviceCluster
    - istio-pilot
    - --templateFile
    - /etc/istio/proxy/envoy_pilot.yaml.tmpl
    - --controlPlaneAuthPolicy
    - MUTUAL_TLS # Tells Pilot to use mTLS
    volumeMounts:
    - mountPath: /etc/certs
      name: istio-certs
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: istio-pilot-service-account-token-s8ndg
      readOnly: true
  serviceAccount: istio-pilot-service-account
  serviceAccountName: istio-pilot-service-account
  volumes:
  - name: istio-certs
    secret:
      defaultMode: 420
      secretName: istio.istio-pilot-service-account
  - name: istio-pilot-service-account-token-s8ndg
    secret:
      defaultMode: 420
      secretName: istio-pilot-service-account-token-s8ndg

Now use base64 and openssl to check the certificate validity of each Istio control-plane service. First, obtain the list of secrets used by the Istio control-plane services:

export SECRETS=$(kubectl get secret -n istio-system | grep "istio.io/key-and-cert" | cut -d " " -f 1)

Next, check the certificate start/end dates for each Istio control-plane service:

for s in $SECRETS; do kubectl get secret/$s -n istio-system -o "jsonpath={.data['cert-chain\.pem']}" \
| base64 --decode >> $s-certchain.pem && openssl x509 -noout -in $s-certchain.pem \
-subject -issuer -dates && rm -rf $s-certchain.pem; done

You should see the following output for each service in $SECRETS, indicating the notBefore and notAfter dates for validating each certificate:

subject= /O=
issuer= /O=k8s.cluster.local
notBefore=Jul 24 15:35:30 2018 GMT
notAfter=Oct 22 15:35:30 2018 GMT

By default, certificates are valid for 90 days. Citadel uses the max-workload-cert-ttl and workload-cert-ttl configuration flags to control this behavior. Both flags default to 90 days. The workload-cert-ttl value must be less than the max-workload-cert-ttl value. Otherwise, Citadel will fail issuing certificates.

Check the validity of the CA certificate used by Citadel:

kubectl get secret/istio-ca-secret -n istio-system -o "jsonpath={.data['ca-cert\.pem']}" \
| base64 --decode >> ca-cert.pem && openssl x509 -noout -in ca-cert.pem \
-subject -issuer -dates && rm -rf ca-cert.pem

You should see an output similar to the other Istio services. However, the Citadel CA certificate is valid for 365 days:

subject= /O=k8s.cluster.local
issuer= /O=k8s.cluster.local
notBefore=Jul 24 15:35:26 2018 GMT
notAfter=Jul 24 15:35:26 2019 GMT

You have now completed verifying the Istio control-plane certificates.

Istio Data-Plane Verification

End-user services running in an Istio mesh can be secured without making any modifications to the application code. This is accomplished in the same manner that Istio control-plane services are secured. Deploy the bookinfo sample application that will be used for testing. Verify the bookinfo application is running:

kubectl get po
NAME                                                        READY     STATUS             RESTARTS   AGE
<SNIP>
details-v1-7b97668445-fdxvn                                 2/2       Running            0          13h
productpage-v1-7bbdd59459-qmngl                             2/2       Running            0          13h
ratings-v1-76dc7f6b9-xmdpp                                  2/2       Running            0          1d
reviews-v1-64545d97b4-wqcpm                                 2/2       Running            0          1d
reviews-v2-8cb9489c6-rqnl2                                  2/2       Running            0          13h
reviews-v3-6bc884b456-7896d                                 2/2       Running            0          1d

Verify that mTLS is enabled by viewing the istio configmap details:

$ kubectl get cm/istio -n istio-system -o yaml | grep MUTUAL_TLS
    authPolicy: MUTUAL_TLS
      controlPlaneAuthPolicy: MUTUAL_TLS

authPolicy: MUTUAL_TLS instructs the Istio proxies to use mTLS for secure communication within the service mesh. You can verify this by viewing the configuration of a sidecar proxy for a bookinfo service:

$ kubectl exec details-v1-7b97668445-fdxvn -c istio-proxy cat /etc/istio/proxy/envoy-rev0.json
{
  <SNIP>
  "static_resources": {
    "clusters": [
    {
    "name": "xds-grpc",
    "type": "STRICT_DNS",
    "connect_timeout": {"seconds": 10, "nanos": 0},
    "lb_policy": "ROUND_ROBIN",

      "tls_context": {
        "common_tls_context": {
          "alpn_protocols": "h2",
          "tls_certificates": {
            "certificate_chain": {
              "filename": "/etc/certs/cert-chain.pem"
            },
            "private_key": {
              "filename": "/etc/certs/key.pem"
            }
          },
          "validation_context": {
            "trusted_ca": {
              "filename": "/etc/certs/root-cert.pem"
            },
            "verify_subject_alt_name": [
              "spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account"
            ]
          }
        }
      },
<SNIP>
}

As with the Istio control-plane, the certificate files are made available through a Kubernetes secret. The secret data (i.e. certificate files) get mounted as a volume into the proxy container. End-user services within the same Kubernetes namespace share a common certificate. You can view the details of the secret that holds the certificate used by the bookinfo services with the following command:

kubectl get secret/istio.default -o yaml
apiVersion: v1
data:
  cert-chain.pem: REDACTED
  key.pem: REDACTED
  root-cert.pem: REDACTED
kind: Secret
metadata:
  annotations:
    istio.io/service-account.name: default
  creationTimestamp: 2018-07-24T15:12:44Z
  name: istio.default
  namespace: default
  resourceVersion: "106956"
  selfLink: /api/v1/namespaces/default/secrets/istio.default
  uid: 041a3566-8f54-11e8-aaf2-00505693a589
type: istio.io/key-and-cert

Now use base64 and openssl to check the certificate validity of the istio.default secret:

kubectl get secret/istio.default -n istio-system -o "jsonpath={.data['cert-chain\.pem']}" \
| base64 --decode >> default-certchain.pem && openssl x509 -noout -in default-certchain.pem \
-subject -issuer -dates && rm -rf default-certchain.pem

You should see the following output, indicating the starting and ending dates for considering the certificate valid:

subject= /O=
issuer= /O=k8s.cluster.local
notBefore=Jul 24 15:35:30 2018 GMT
notAfter=Oct 22 15:35:30 2018 GMT

You have now completed verifying the Istio data-plane certificates.

danehans/CT1611.md

Introduction

Prerequisites

Istio Control-Plane Verification

Istio Data-Plane Verification