go dumps for Rancher 2.x support

README.md

go dumps for Rancher 2.x support

If you are experiencing random crashing of your Rancher 2.x pods or docker container, sometimes Rancher support will ask you to take a go routine dump or a memory dump. Below are the commmands you need to run inside of your Rancher container or pod.

#exec into Rancher pod or container first
mkdir dumps
curl localhost:6060/debug/pprof/goroutine -o dumps/goroutine
curl localhost:6060/debug/pprof/heap -o dumps/heap
curl localhost:6060/debug/pprof/threadcreate -o dumps/threadcreate
curl localhost:6060/debug/pprof/block -o dumps/block
curl localhost:6060/debug/pprof/mutex -o dumps/mutex
tar -zcf dumps.tar.gz dumps
#exit container
exit
#copy file out of container
docker cp <id>:/var/lib/rancher/dumps.tar.gz /tmp/

Single server rancher k3s profile dump

docker exec -i CONTAINER kubectl get --raw /debug/pprof/profile > profile
docker exec -i $CONTAINER kubectl get --raw /debug/pprof/goroutine > goroutine
docker exec -i $CONTAINER kubectl get --raw /debug/pprof/heap > heap
docker exec -i $CONTAINER kubectl get --raw /debug/pprof/threadcreate > threadcreate
docker exec -i $CONTAINER kubectl get --raw /debug/pprof/block > block
docker exec -i $CONTAINER kubectl get --raw /debug/pprof/mutex > mutex

Deprecate use of this script in favor for:

for pod in $(kubectl get pods --no-headers -l app=rancher | cut -d ' ' -f1); do
  echo getting heap for $pod
  kubectl exec $pod -- curl -s http://localhost:6060/debug/pprof/heap -o heap;
  kubectl cp $pod:heap $pod-heap;
  echo saved heap $pod-heap
done```