kong-mesh-tracing.md

Kong Mesh Tracing

This document shows how to use distributed tracing in Kong Mesh while running on OpenShift.

Environment

Client Version: 4.10.4
Server Version: 4.9.9
Kubernetes Version: v1.22.3+4dd1b5a
Client: Kong Mesh 1.7.0

TOC

• Install Kong Mesh
• Deploy Demo App
• Deploy Metrics
• Distributed Tracing
• Cleanup

Install Kong Mesh

Steps are found in the Kong Mesh Documentation

1. Download Kong Mesh

curl -L https://docs.konghq.com/mesh/installer.sh | sh -
cd kong-mesh-1.7.0/bin
mv kumactl /usr/local/bin/kumactl
cd ../..
rm -rf kong-mesh-1.7*

2. Install Kong Mesh

kumactl install control-plane --cni-enabled | oc apply -f -

3. Make sure control plane pod has come up

kubectl get po -n kong-mesh-system 

4. Make sure you can access the Control Plane Dashboard

kubectl port-forward svc/kong-mesh-control-plane -n kong-mesh-system 5681

go to http://localhost:5681/gui

Deploy Demo App

Kong Mesh explicitly specifies a UID for the kuma-dp sidecar to avoid capturing traffic from kuma-dp itself. You must grant nonroot Security Context Constraint to the application namespace.

oc adm policy add-scc-to-group nonroot system:serviceaccounts:kuma-demo

Install the demo app

kubectl create -f https://raw.githubusercontent.com/kumahq/kuma-counter-demo/master/demo.yaml

Port-forward the demo-app service to localhost 5000

kubectl port-forward svc/demo-app -n kuma-demo 5000

Go to localhost:5000

Deploy Metrics

This is where we diverge from the beaten path of the documentation. When we install metrics, the grafana pod has an initContainer that uses wget to pull down dashboards and datasources. THIS WILL NOT WORK WITH SIDECAR INJECTION ENABLED. For more detailed information regarding the problem, look at Problem Statement

1. Add nonroot Security Context Constraint to the kong-mesh-metrics service account.

oc adm policy add-scc-to-group nonroot system:serviceaccounts:kong-mesh-metrics

2. Enable Traffic Metrics (Install Prometheus and Grafana )

kumactl install metrics | kubectl apply -f -

3. Remove sidecar injection label from kong-mesh-metrics namespace

kubectl label ns kong-mesh-metrics kuma.io/sidecar-injection-

4. Enable metrics on our Mesh Object

echo "apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
    - name: ca-1
      type: builtin
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus
      conf:
        skipMTLS: true" | kubectl apply -f -

5. Scale the pods to 0 and then to 1 Replica to avoid collisions

kubectl scale deploy -n kong-mesh-metrics --replicas=0 --all

kubectl delete po -n kong-mesh-metrics --force --grace-period=0 --all 

kubectl scale deploy -n kong-mesh-metrics --replicas=1 --all

6. Generate Traffic in the mesh Port-forward the demo-app service to localhost 5000

kubectl port-forward svc/demo-app -n kuma-demo 5000

Go to localhost:5000 and click "Auto Increment"

7. Port-forward the Grafana service to 3000

k port-forward svc/grafana -n kong-mesh-metrics 3000:80 

go to http://localhost:3000/d/z6C1v-NGk/kuma-cp?orgId=1&refresh=5s

Now you can see the scrape process occuring, the link above will show you the Kuma Control Plane Dashboard. We now have happy Prometheus and Grafana instances with sufficient permissions to do their jobs but not excessive.

Distributed Tracing

Kong supports distributed tracing with zipkin, and datadog

1. Deploy Jeager

kumactl install tracing | kubectl apply -f -

2. Make sure the pod is up in kong-mesh-tracing

kubectl get po -n kong-mesh-tracing

3. Enable metrics on our Mesh Object

echo "apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
    - name: ca-1
      type: builtin
  tracing:
    defaultBackend: jaeger-collector
    backends:
    - name: jaeger-collector
      type: zipkin
      sampling: 100.0
      conf:
        url: http://jaeger-collector.kong-mesh-tracing:9411/api/v2/spans
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus
      conf:
        skipMTLS: true" | kubectl apply -f -

4. Add TrafficTrace resource

kubectl apply -f -<<EOF
apiVersion: kuma.io/v1alpha1
kind: TrafficTrace
mesh: default
metadata:
  name: trace-all-traffic
spec:
  selectors:
  - match:
      kuma.io/service: '*'
      app: demo
  conf:
    backend: jaeger-collector # or the name of any backend defined for the mesh 
EOF

5. Configure Grafana to Visualize the logs

k port-forward svc/grafana -n kong-mesh-metrics 3000:80 

go to http://localhost:3000/ and configure a new datasource with url=http://jaeger-collector.kong-mesh-tracing/ so Grafana will be able to retrieve the traces from Jaeger.

• click gear icon on left panel
• click Jaeger
• edit url http://jaeger-query.kong-mesh-tracing
• save & test

6. Pull Up Jaeger and Look at Traces

k port-forward svc/jaeger-query -n kong-mesh-tracing 3001:80

go to http://localhost:3001

Cleanup

# Delete the Mesh Object
kubectl delete mesh default

# Delete TrafficTrace
kubectl delete traffictrace default 

# Delete Metrics
kumactl install metrics | kubectl delete -f -


# Delete Tracing 
kumactl install tracing | kubectl delete -f -


# Delete Control Plane
kumactl install control-plane --cni-enabled | oc delete -f -

# Delete Demo Apps
kubectl delete -f https://raw.githubusercontent.com/kumahq/kuma-counter-demo/master/demo.yaml