pprof

With a single import _ "net/http/pprof" you can add profiling endpoints to a HTTP server.

package main

import (
	"fmt"
	"log"
	"net/http"
	_ "net/http/pprof" // register debug routes on default mux
)

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Hello World!")
	})
	log.Fatal(http.ListenAndServe(":8080", nil))
}

This will now report diagnostics via /debug/pprof/.

/debug/pprof/profile: 30-second CPU profile /debug/pprof/heap: heap profile
/debug/pprof/goroutine?debug=1: all goroutines with stack traces
/debug/pprof/trace: take a trace

Risks

This mechanism is dangerously simple. It only requires one import which could be anywhere!

Security issues:

Function names and file paths are revealed.
Profiling data may reveal business sensitive information (for example traffic to a web server)
Profiling degrades performance, providing a vector for a DoS attack

Depending on your application, leaving a debugging server may not necessarily be a critical security hole. At a minimum it’s inadvisable, but it could be much worse.

Access Control

A simple and effective option is to put the pprof http server on a separate port on localhost, separate from the application http server. If the application does not use the http default multiplexer, starting the profiling http server is as simple as:

	go func() {
		log.Println(http.ListenAndServe("localhost:8081"))
	}()

Otherwise, you can run the following additional setup prior to any http.HandleFunc() calls:

	// Save pprof handlers first.
	pprofMux := http.DefaultServeMux
	http.DefaultServeMux = http.NewServeMux()

	// Pprof server.
	go func() {
		log.Println(http.ListenAndServe("localhost:8081", pprofMux))
	}()

	// Application server.
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "Hello World!")
	})
	log.Fatal(http.ListenAndServe(":8080", nil))

Either of the above snippets can be conditionally run, so the profiling server may be turned on or off by command line flags or application configuration. However, since they are now only locally accessible, there is no downside to leaving them on.

Remote Control

Now that you have the profiling hooks safely exposed via a localhost-only interface, you can invoke the following:

$ go tool pprof http://localhost:8081/debug/pprof/profile

However, since go compiles to a static binary which can be installed without any go-related dependencies, there is a possibility that you don't have go tools where the program is running. SSH port forwarding(AKA SSH tunneling) can be used to make go tool pprof on your local machine profile the remote code over the SSH tunnel.

$ ssh -fNT -L 8081:localhost:8081 -i ~/creds/my-key.pem [email protected]

$ go tool pprof -inuse_objects /path/to/binary/myapp http://localhost:8081/debug/pprof/heap
Fetching profile over HTTP from http://localhost:8081/debug/pprof/heap
Saved profile in /home/suhaskaranth/pprof/pprof.myapp.alloc_objects.alloc_space.inuse_objects.inuse_space.010.pb.gz
File: myapp
Type: inuse_objects
Time: Aug 11, 2018 at 11:45am (IST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 65536, 99.73% of 65714 total
Dropped 50 nodes (cum <= 328)
Showing top 10 nodes out of 20
      flat  flat%   sum%        cum   cum%
     32768 49.86% 49.86%      65536 99.73%  myapp/vendor/github.com/ugorji/go/codec.init.0
     32768 49.86% 99.73%      32768 49.86%  sync.(*Map).Range
         0     0% 99.73%      32768 49.86%  encoding/gob.(*Decoder).decOpFor
         0     0% 99.73%      32768 49.86%  encoding/gob.(*Decoder).decodeInterface
         0     0% 99.73%      32768 49.86%  encoding/gob.decInt32Slice
         0     0% 99.73%      32768 49.86%  encoding/gob.decInt64Slice
         0     0% 99.73%      32768 49.86%  encoding/gob.decStringSlice
         0     0% 99.73%      32768 49.86%  myapp/adapter/internal.SendNotifications
         0     0% 99.73%      32768 49.86%  myapp/cache.prepStaticComponent
         0     0% 99.73%      32768 49.86%  myapp/model/redis.(*Client).Del
(pprof) quit

To stop the SSH session running in the background, check the running processes for the specified port:

$ ps -aux | grep 8081
suhaska+ 23340  0.0  0.0  52516   752 ?        Ss   16:02   0:00 ssh -fNT -L 8081:localhost:8081 -i ~/creds/my-key.pem [email protected]
suhaska+ 23386  0.0  0.0  14432  1052 pts/0    S+   16:02   0:00 grep --color=auto 8081

$ kill -QUIT 23340

sudo-suhas/pprof.md

pprof

Risks

Access Control

Remote Control

References