Kong Mesh Metrics

kong-mesh-metrics.md

Kong Mesh Metrics

This document shows how to work around the Metrics Problem in Kong Mesh while running on OpenShift.

Environment

Client Version: 4.10.4
Server Version: 4.9.9
Kubernetes Version: v1.22.3+4dd1b5a
Client: Kong Mesh 1.7.0

TOC

• Install Kong Mesh
• Deploy Demo App
• Deploy Metrics
• Problem Statement
• Cleanup

Install Kong Mesh

Steps are found in the Kong Mesh Documentation

1. Download Kong Mesh

curl -L https://docs.konghq.com/mesh/installer.sh | sh -
cd kong-mesh-1.7.0/bin
mv kumactl /usr/local/bin/kumactl
cd ../..
rm -rf kong-mesh-1.7*

2. Install Kong Mesh

kumactl install control-plane --cni-enabled | oc apply -f -

3. Make sure control plane pod has come up

kubectl get po -n kong-mesh-system 

4. Make sure you can access the Control Plane Dashboard

kubectl port-forward svc/kong-mesh-control-plane -n kong-mesh-system 5681

go to http://localhost:5681/gui

Deploy Demo App

Kong Mesh explicitly specifies a UID for the kuma-dp sidecar to avoid capturing traffic from kuma-dp itself. You must grant nonroot Security Context Constraint to the application namespace.

oc adm policy add-scc-to-group nonroot system:serviceaccounts:kuma-demo

Install the demo app

kubectl create -f https://raw.githubusercontent.com/kumahq/kuma-counter-demo/master/demo.yaml

Port-forward the demo-app service to localhost 5000

kubectl port-forward svc/demo-app -n kuma-demo 5000

Go to localhost:5000

Deploy Metrics

This is where we diverge from the beaten path of the documentation. When we install metrics, the grafana pod has an initContainer that uses wget to pull down dashboards and datasources. THIS WILL NOT WORK WITH SIDECAR INJECTION ENABLED. For more detailed information regarding the problem, look at Problem Statement

1. Add nonroot Security Context Constraint

oc adm policy add-scc-to-group nonroot system:serviceaccounts:kong-mesh-metrics

2. Enable Traffic Metrics (Install Prometheus and Grafana )

kumactl install metrics | kubectl apply -f -

3. Remove sidecar injection label from kong-mesh-metrics namespace

kubectl label ns kong-mesh-metrics kuma.io/sidecar-injection-

4. Enable metrics on our Mesh Object

echo "apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
    - name: ca-1
      type: builtin
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus
      conf:
        skipMTLS: true" | kubectl apply -f -

5. Scale the pods to 0 and then to 1 Replica to avoid collisions

kubectl scale deploy -n kong-mesh-metrics --replicas=0 --all

kubectl delete po -n kong-mesh-metrics --force --grace-period=0 --all 

kubectl scale deploy -n kong-mesh-metrics --replicas=1 --all

6. Generate Traffic in the mesh Port-forward the demo-app service to localhost 5000

kubectl port-forward svc/demo-app -n kuma-demo 5000

Go to localhost:5000 and click "Auto Increment"

7. Port-forward the Grafana service to 3000

k port-forward svc/grafana -n kong-mesh-metrics 3000:80 

go to localhost:3000

Now you can see the scrape process occuring, the link above will show you the Kuma Control Plane Dashboard. We now have happy Prometheus and Grafana instances with sufficient permissions to do their jobs but not excessive.

Problem Statement

This is a deep dive on the current problem with the metrics. The problem is that the initContainer in the is unable to execute the wget command.

Below of is the description of the pod in question showing a failed initContainer.

# k describe po -n kong-mesh-metrics -l app=grafana

Init Containers:
  init-plugins:
    Container ID:  cri-o://06424b2655e8710db9e984e0e02ad4b60df4295b96b07dbe02a4a0a96085a531
    Image:         alpine
    Image ID:      docker.io/library/alpine@sha256:4edbd2beb5f78b1014028f4fbb99f3237d9561100b6881aabbf5acce2c4f9454
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      wget -O /tmp/kuma.zip https://github.com/kumahq/kuma-grafana-datasource/releases/download/v0.1.0/kumahq-kuma-datasource-0.1.0.zip && unzip /tmp/kuma.zip -d /var/lib/grafana/plugins/ && rm /tmp/kuma.zip
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error

Here we exec into the container and find that github.com is not being resolved. We changed the initContainer to sleep first

kubectl exec -it grafana-7bb6f49d77-lq4vd -n kong-mesh-metrics -c init-plugins -- wget -O /tmp/kuma.zip https://github.com/kumahq/kuma-grafana-datasource/releases/download/v0.1.0/kumahq-kuma-datasource-0.1.0.zip && unzip /tmp/kuma.zip -d /var/lib 
/grafana/plugins/ && rm /tmp/kuma.zip
wget: bad address 'github.com'

Cleanup

# Delete the Mesh Object
kubectl delete mesh default

# Delete Metrics
kumactl install metrics | kubectl delete -f -

# Delete Control Plane
kumactl install control-plane --cni-enabled | oc delete -f -

# Delete Demo Apps
kubectl delete -f https://raw.githubusercontent.com/kumahq/kuma-counter-demo/master/demo.yaml