Dump gardenlet profile

README.md

Dump gardenlet profile

Use the following manifest to dump the {heap,allocs,goroutine} profile of a running gardenlet every 10 minutes:

$ k apply -f profile-dump.yaml
deployment.apps/profile-dump created
persistentvolumeclaim/profile-dump created

$ kg get po,pvc -l app=profile-dump
NAME                                READY   STATUS    RESTARTS   AGE
pod/profile-dump-58d456548b-csswx   1/1     Running   0          43s

NAME                                 STATUS   VOLUME                                            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/profile-dump   Bound    pv-foo-bar-0398ba56-dca7-467c-a525-17a2365fae1d   5Gi        RWO            default        43s

$ kg logs -f profile-dump-58d456548b-csswx
====== Tue Sep 14 09:33:05 UTC 2021 ======
Connecting to gardenlet.garden.svc:443 (10.243.45.107:443)
saving to '/var/profile-dump/gardenlet-20210914T093305Z-heap'
gardenlet-20210914T0 100% |********************************|  157k  0:00:00 ETA
'/var/profile-dump/gardenlet-20210914T093305Z-heap' saved
Connecting to gardenlet.garden.svc:443 (10.243.45.107:443)
saving to '/var/profile-dump/gardenlet-20210914T093305Z-allocs'
gardenlet-20210914T0 100% |********************************|  157k  0:00:00 ETA
'/var/profile-dump/gardenlet-20210914T093305Z-allocs' saved
Connecting to gardenlet.garden.svc:443 (10.243.45.107:443)
saving to '/var/profile-dump/gardenlet-20210914T093305Z-goroutine'
gardenlet-20210914T0 100% |********************************|  7274  0:00:00 ETA
'/var/profile-dump/gardenlet-20210914T093305Z-goroutine' saved
...

Retrieve the profiles via kubectl cp and start pprof:

$ k cp garden/profile-dump-58d456548b-csswx:/var/profile-dump /tmp/profile-dump
tar: removing leading '/' from member names

$ go tool pprof -http :8080 /tmp/profile-dump/gardenlet-20210914T093305Z-heap
Serving web UI on http://localhost:8080

Cleanup:

$ k delete -f profile-dump.yaml
deployment.apps "profile-dump" deleted
persistentvolumeclaim "profile-dump" deleted

profile-dump.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: profile-dump
  namespace: garden
spec:
  selector:
    matchLabels:
      app: profile-dump
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: profile-dump
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: profile-dump
        image: eu.gcr.io/gardener-project/3rd/alpine:3.13.5
        command:
        - sh
        - -c
        - |
          while true ; do
            timestamp="$(date --utc "+%Y%m%dT%H%M%SZ")"
            echo "====== $timestamp ======"
            for p in heap allocs goroutine ; do
              wget --tries=5 --no-check-certificate https://gardenlet.garden.svc:443/debug/pprof/$p -O /var/profile-dump/gardenlet-$timestamp-$p
            done
            sleep 600 # 10m
          done
        volumeMounts:
        - name: profile-dump
          mountPath: /var/profile-dump
      volumes:
      - name: profile-dump
        persistentVolumeClaim:
          claimName: profile-dump
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: profile-dump
  namespace: garden
  labels:
    app: profile-dump
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: default
  resources:
    requests:
      storage: 5Gi