本文转发自以下链接: https://github.com/istio/istio/wiki/Analyzing-Istio-Performance#memory
Analyzing Istio Performance
Control Plane
Most control plane components are instrumented with pprof. This allows profiling of memory, CPU usage, etc.
To profile Pilot, follow these steps:
All of these steps assume Pilot is located at localhost:8080
. For most situations, this can be achieved with kubectl port-forward -n istio-system PILOT-POD 8080
.
More details about pprof can be found here.
Memory
To profile memory usage:
$ go tool pprof -http=:8888 localhost:8080/debug/pprof/heap
Fetching profile over HTTP from http://localhost:8080/debug/pprof/heap
Saved profile in /home/pprof/pprof.pilot-discovery.alloc_objects.alloc_space.inuse_objects.inuse_space.032.pb.gz
Running this will fetch a memory profile from Pilot, open up a web UI, and save the profile to a gz
file.
On the web UI (localhost:8888
in this example), you can view current memory and total allocated memory, in different formats. Generally the "Flame Graph" view is the easiest to understand.
CPU
To profile CPU:
$ go tool pprof -http=:8888 localhost:8080/debug/pprof/profile
Note: whereas memory will capture a snapshot of current usage and all lifetime allocations, this will poll for 30s and capture CPU usage during this time.
Goroutines
To debug a goroutine leak or deadlock, it can be useful to dump all active goroutines. This can be done with
curl 'http://localhost:8080/debug/pprof/goroutine?debug=2'
Note: output may be very large
Data Plane
Prerequisite
(For Istio 1.5 and older) In order to write heap profiles, Envoy needs to be able to write to the file system. By default, this is restricted by readOnlyRootFilesystem: true
. This can be change in a few ways:
- Install with
--set values.global.proxy.enableCoreDump=true
- Manually modify
istio-sidecar-injector
configmap - Do a manual injection and remove
readOnlyRootFilesystem: true
Profile
On Istio 1.5 and older:
export POD=pod-name
export NS=istio-system
kubectl exec -n "$NS" "$POD" -c istio-proxy -- sh -c 'sudo mkdir -p /var/log/envoy && sudo chmod 777 /var/log/envoy && curl -X POST -s "http://localhost:15000/heapprofiler?enable=y"'
sleep 15
kubectl exec -n "$NS" "$POD" -c istio-proxy -- sh -c 'curl -X POST -s "http://localhost:15000/heapprofiler?enable=n"'
rm -rf /tmp/envoy
kubectl cp -n "$NS" "$POD":/var/log/envoy/ /tmp/envoy -c istio-proxy
kubectl cp -n "$NS" "$POD":/lib/x86_64-linux-gnu /tmp/envoy/lib -c istio-proxy
kubectl cp -n "$NS" "$POD":/usr/local/bin/envoy /tmp/envoy/lib/envoy -c istio-proxy
On Istio 1.6+
export POD=pod-name
export NS=istio-system
export PROFILER="cpu" # Can also be "heap", for a heap profile
kubectl exec -n "$NS" "$POD" -c istio-proxy -- curl -X POST -s "http://localhost:15000/${PROFILER}profiler?enable=y"
sleep 15
kubectl exec -n "$NS" "$POD" -c istio-proxy -- curl -X POST -s "http://localhost:15000/${PROFILER}profiler?enable=n"
rm -rf /tmp/envoy
kubectl cp -n "$NS" "$POD":/var/lib/istio/data /tmp/envoy -c istio-proxy
kubectl cp -n "$NS" "$POD":/lib/x86_64-linux-gnu /tmp/envoy/lib -c istio-proxy
kubectl cp -n "$NS" "$POD":/usr/local/bin/envoy /tmp/envoy/lib/envoy -c istio-proxy
Visualize profile pprof installation
Install pprof, then run:
PPROF_BINARY_PATH=/tmp/envoy/lib/ pprof -pdf /tmp/envoy/lib/envoy /tmp/envoy/envoy.prof
Or, interactively
PPROF_BINARY_PATH=/tmp/envoy/lib/ pprof /tmp/envoy/lib/envoy /tmp/envoy/envoy.prof
Or, through the web UI
PPROF_BINARY_PATH=/tmp/envoy/lib/ pprof -http=localhost:8000 /tmp/envoy/lib/envoy /tmp/envoy/envoy.prof