istio pilot pprof

Introduction

Istio Pilot provides pprof http endpoints for profiling the application. The --profile flag is used to enable Pilot's pprof http endpoints. This flag is enabled by default.

Prerequisites

You must have a functioning Istio mesh running on a Kubernetes cluster.
Go tools must be installed on your client.

Create the Ingress resource.

Create an Ingress for the Istio Pilot service. Note: Ingress resources require an Ingress Controller.

cat <<EOF | kubectl create -n istio-system -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: pilot-pprof
  annotations:
    kubernetes.io/ingress.class: istio
spec:
  rules:
  - http:
      paths:
      - path: /debug/pprof/.*
        backend:
          serviceName: istio-pilot
          servicePort: 15003
EOF

Get the IP address of the Ingress resource:

$ PPROF_INGRESS_IP=$(kubectl -n istio-system get ing/pilot-pprof -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Run the go pprof tool:

$ go tool pprof http://$PPROF_INGRESS_IP/debug/pprof/heap
Fetching profile over HTTP from http://35.203.174.121/debug/pprof/heap
Local symbolization failed for pilot-discovery: open /usr/local/bin/pilot-discovery: no such file or directory
Some binary filenames not available. Symbolization may be incomplete.
Try setting PPROF_BINARY_PATH to the search path for local binaries.
Saved profile in /Users/daneyonhansen/pprof/pprof.pilot-discovery.alloc_objects.alloc_space.inuse_objects.inuse_space.005.pb.gz
File: pilot-discovery
Build ID: f7bf4798c7e75f0e85996bcf87cee8385ae771c4
Type: inuse_space
Time: Feb 19, 2018 at 10:50am (PST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)

Use any of the pprof commands. Here is an example of using the top10 command to show the top 10 samples in the profile:

top10
Showing nodes accounting for 3905.98kB, 100% of 3905.98kB total
Showing top 10 nodes out of 68
      flat  flat%   sum%        cum   cum%
 1024.56kB 26.23% 26.23%  1024.56kB 26.23%  reflect.mapassign /builder/home/.cache/bazel/_bazel_root/b6e836c920560cea0d05d0eeb2455c03/external/go_sdk/src/runtime/hashmap.go
  809.97kB 20.74% 46.97%   809.97kB 20.74%  golang.org/x/net/http2.(*ClientConn).frameScratchBuffer external/org_golang_x_net/http2/transport.go
  532.26kB 13.63% 60.59%   532.26kB 13.63%  bytes.makeSlice /builder/home/.cache/bazel/_bazel_root/b6e836c920560cea0d05d0eeb2455c03/external/go_sdk/src/bytes/buffer.go
     514kB 13.16% 73.75%      514kB 13.16%  bufio.NewWriterSize /builder/home/.cache/bazel/_bazel_root/b6e836c920560cea0d05d0eeb2455c03/external/go_sdk/src/bufio/bufio.go
     513kB 13.13% 86.89%      513kB 13.13%  regexp.onePassCopy /builder/home/.cache/bazel/_bazel_root/b6e836c920560cea0d05d0eeb2455c03/external/go_sdk/src/regexp/onepass.go
  512.19kB 13.11%   100%   512.19kB 13.11%  github.com/gogo/protobuf/proto.getPropertiesLocked external/com_github_gogo_protobuf/proto/properties.go
         0     0%   100%   532.26kB 13.63%  bytes.(*Buffer).WriteByte /builder/home/.cache/bazel/_bazel_root/b6e836c920560cea0d05d0eeb2455c03/external/go_sdk/src/bytes/buffer.go
         0     0%   100%   532.26kB 13.63%  bytes.(*Buffer).grow /builder/home/.cache/bazel/_bazel_root/b6e836c920560cea0d05d0eeb2455c03/external/go_sdk/src/bytes/buffer.go
         0     0%   100%  1024.56kB 26.23%  encoding/json.(*decodeState).object /builder/home/.cache/bazel/_bazel_root/b6e836c920560cea0d05d0eeb2455c03/external/go_sdk/src/encoding/json/decode.go
         0     0%   100%  1024.56kB 26.23%  encoding/json.(*decodeState).unmarshal /builder/home/.cache/bazel/_bazel_root/b6e836c920560cea0d05d0eeb2455c03/external/go_sdk/src/encoding/json/decode.go

(Optional) Access Pilot pprof from a Browser

You can also access the Istio Pilot pprof endpoint through your browser: http://$PPROF_INGRESS_IP/debug/pprof/