How to run go pprof on Docker daemon

pprof-in-docker-daemon.md

#Use pprof debug docker daemon

• Use pprof debug docker daemon
• pprof debug entrypoint
• Start docker daemon in debug mode
• Run socat to make docker sock available via tcp port
• Access debug url entrypoint
  • show global vars
  • get command line
  • run pprof on your client:
  • generate graph
  • get symbol
  • other

pprof debug entrypoint

source file: docker\api\server\profiler.go

const debugPathPrefix = "/debug/"

func profilerSetup(mainRouter *mux.Router) {
	var r = mainRouter.PathPrefix(debugPathPrefix).Subrouter()
	r.HandleFunc("/vars", expVars)
	r.HandleFunc("/pprof/", pprof.Index)
	r.HandleFunc("/pprof/cmdline", pprof.Cmdline)
	r.HandleFunc("/pprof/profile", pprof.Profile)
	r.HandleFunc("/pprof/symbol", pprof.Symbol)
	r.HandleFunc("/pprof/trace", pprof.Trace)
	r.HandleFunc("/pprof/block", pprof.Handler("block").ServeHTTP)
	r.HandleFunc("/pprof/heap", pprof.Handler("heap").ServeHTTP)
	r.HandleFunc("/pprof/goroutine", pprof.Handler("goroutine").ServeHTTP)
	r.HandleFunc("/pprof/threadcreate", pprof.Handler("threadcreate").ServeHTTP)
}

Start docker daemon in debug mode

add -D to command line

$ /usr/bin/docker daemon -D -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock

Run socat to make docker sock available via tcp port

note the IP to listen at

$ socat -d -d TCP-LISTEN:8080,fork,bind=192.168.1.137 UNIX:/var/run/docker.sock
  2016/01/23 12:59:43 socat[3113] N listening on AF=2 192.168.1.137:8080

Access debug url entrypoint

show global vars

$ curl -s http://192.168.1.137:8080/debug/vars | jq .
  {
    "cmdline": [
      "/usr/bin/docker",
      "daemon",
      "-D",
      "-H",
      "tcp://0.0.0.0:2375",
      "-H",
      "unix:///var/run/docker.sock",
      "-api-enable-cors",
      "--storage-driver=aufs",
      "--insecure-registry",
      "registry.hyper.sh:5000",
      "--insecure-registry",
      "192.168.1.137:5000"
    ],
    "memstats": {
      "Alloc": 7140904,
      "TotalAlloc": 20379824,
      "Sys": 14559480,
      "Lookups": 3090,
      "Mallocs": 313164,
      "Frees": 199260,
      "HeapAlloc": 7140904,
  ...

get command line

$ curl -s http://192.168.1.137:8080/debug/pprof/cmdline
  /usr/bin/docker daemon -D -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock -api-enable-cors --storage-driver=aufs --insecure-registry registry.hyper.sh:5000 --insecure-registry 192.168.1.137:5000

run pprof on your client:

$ go tool pprof http://192.168.1.137:8080/debug/pprof/profile
  Fetching profile from http://192.168.1.137:8080/debug/pprof/profile
  Please wait... (30s)
  Saved profile in /home/xjimmy/pprof/pprof.192.168.1.137:8080.samples.cpu.001.pb.gz
  Entering interactive mode (type "help" for commands)
  (pprof) web     
  (pprof) web ploop
  (pprof)

generate graph

$ cd /home/xjimmy/pprof
$ gunzip pprof.192.168.1.137:8080.samples.cpu.001.pb.gz
$ go tool pprof --pdf pprof.dockerd.192.168.1.137:8080.samples.cpu.001.pb > callgraph.pdf

get symbol

$ curl -s http://192.168.1.137:8080/debug/pprof/symbol
  num_symbols: 1

other

$ culr -s http://192.168.1.137:8080/debug/pprof/block
$ curl -s http://192.168.1.137:8080/debug/pprof/heap
$ curl -s http://192.168.1.137:8080/debug/pprof/goroutine
$ curl -s http://192.168.1.137:8080/debug/pprof/threadcreate

curl can hit unix sockets directly now (since 7.40), e.g. curl --unix-socket /var/run/docker.sock http://_/debug/vars (since 7.50 you have to give curl a hostname, but it doesn't matter what it is, so I put _ in)