1
pprof是什么
pprof 是 go 语言自带的基础库,可以分析程序的运行情况,并且提供可视化的功能。它包含两个相关的库:runtime/pprof 和 net/http/pprof , net/http/pprof 可以提供一个 HTTP 接口,获得性能数据,它的底层使用的 runtime/pprof。
2
pprof如何使用
一般如果我们使用 net/http/pprof 并开启了端口监听,如8001端口,我们就可以通过浏览器访问该服务pprof的页面,例如 http://127.0.0.1:8080/debug/pprof/
页面的下方还有各个命令的解释
我们常用的参数为
allocs | 所有过去内存分配的采样 |
goroutine | 所有当前goroutine的堆栈跟踪 |
heap | 对活动对象的内存分配进行采样 |
profile | cpu profile采样 |
3
使用 pprof 分析 kubelet 性能
想要获取 kubelet 的 pprof 数据,首先需要使用kubectl proxy命令启动API server代理
# kubectl proxy --address='0.0.0.0' --accept-hosts='^*$'
Starting to serve on [::]:8001
直接使用 http://127.0.0.1:8080/debug/pprof/ 查看的是 apiserver 的性能指标,查看某个节点的 kubelet 的 url 为:
http://127.0.0.1:8001/api/v1/nodes/${NODENAME}/proxy/debug/pprof/
我们可以通过go tool pprof命令查看heap堆内存分配情况
go tool pprof http://127.0.0.1:8001/api/v1/nodes/192-168-249-10/proxy/debug/pprof/heap
Fetching profile over HTTP from http://127.0.0.1:8001/api/v1/nodes/192-168-249-10/proxy/debug/pprof/heap
Saved profile in /root/pprof/pprof.kubelet.alloc_objects.alloc_space.inuse_objects.inuse_space.004.pb.gz
File: kubelet
Build ID: 8cbf93d0e67344c230e43c23a345eb609612e432
Type: inuse_space
Time: Oct 19, 2022 at 5:11pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)
这时就进入了pprof的交互模式,我们输入top可以查看内存使用量前十的函数,如下所示
(pprof) top
Showing nodes accounting for 7780.25kB, 52.04% of 14950.70kB total
Showing top 10 nodes out of 191
flat flat% sum% cum cum%
1539kB 10.29% 10.29% 1539kB 10.29% k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/libcontainer.newContainerStats
1032.02kB 6.90% 17.20% 1032.02kB 6.90% k8s.io/kubernetes/vendor/github.com/euank/go-kmsg-parser/kmsgparser.(*parser).Parse.func1
1025.78kB 6.86% 24.06% 1025.78kB 6.86% k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypeWithName
1024.62kB 6.85% 30.91% 1024.62kB 6.85% regexp/syntax.(*compiler).inst
561.50kB 3.76% 34.67% 561.50kB 3.76% k8s.io/kubernetes/vendor/golang.org/x/net/html.init
528.17kB 3.53% 38.20% 528.17kB 3.53% k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).Watch.func1
522.06kB 3.49% 41.69% 522.06kB 3.49% k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/filedesc.(*File).initDecls
516.76kB 3.46% 45.15% 516.76kB 3.46% k8s.io/kubernetes/vendor/sigs.k8s.io/json/internal/golang/encoding/json.typeFields
516.64kB 3.46% 48.60% 516.64kB 3.46% io.ReadAll
513.69kB 3.44% 52.04% 513.69kB 3.44% k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/strs.(*Builder).AppendFullName
可见libcontainer.newContainerStats函数占用内存最多,占用了10%的内存。如果是某个进程占用内存很高,就可以以此为依据,排查占用内存最多的几个函数,再根据源码分析具体原因。
刚才分析的是 heap 内存情况,同样可以根据需要分析前面说的所有采样。
#所有过去内存分配的采样
go tool pprof http://127.0.0.1:8001/debug/pprof/allocs
#对活动对象的内存分配进行采样(活动)
go tool pprof http://127.0.0.1:8001/debug/pprof/heap
# 下载 cpu profile,默认从当前开始收集 30s 的 cpu 使用情况,需要等待 30s
go tool pprof http://127.0.0.1:8001/debug/pprof/profile
# wait 120s
go tool pprof http://127.0.0.1:8001/debug/pprof/profile?seconds=120
#导致同步原语阻塞的堆栈跟踪
go tool pprof http://127.0.0.1:8001/debug/pprof/block
#所有当前goroutine的堆栈跟踪
go tool pprof http://127.0.0.1:8001/debug/pprof/goroutine
#争用互斥锁持有者的堆栈跟踪
go tool pprof http://127.0.0.1:8001/debug/pprof/mutex
#当前程序的执行轨迹。
go tool pprof http://127.0.0.1:8001/debug/pprof/trace
例如我们查看 apiserver 的 goroutine
# go tool pprof http://127.0.0.1:8001/debug/pprof/goroutine
Fetching profile over HTTP from http://127.0.0.1:8001/debug/pprof/goroutine
Saved profile in /root/pprof/pprof.kube-apiserver.goroutine.001.pb.gz
File: kube-apiserver
Type: goroutine
Time: Oct 19, 2022 at 5:27pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 1691, 99.53% of 1699 total
Dropped 128 nodes (cum <= 8)
Showing top 10 nodes out of 116
flat flat% sum% cum cum%
1691 99.53% 99.53% 1691 99.53% runtime.gopark
0 0% 99.53% 67 3.94% bufio.(*Reader).Read
0 0% 99.53% 78 4.59% bytes.(*Buffer).ReadFrom
0 0% 99.53% 59 3.47% context.propagateCancel.func1
0 0% 99.53% 78 4.59% crypto/tls.(*Conn).Read
0 0% 99.53% 78 4.59% crypto/tls.(*Conn).readFromUntil
0 0% 99.53% 78 4.59% crypto/tls.(*Conn).readRecord (inline)
0 0% 99.53% 78 4.59% crypto/tls.(*Conn).readRecordOrCCS
0 0% 99.53% 78 4.59% crypto/tls.(*atLeastReader).Read
0 0% 99.53% 78 4.59% internal/poll.(*FD).Read
pprof同时还能够通过 list 命令查看指定函数的具体函数代码
(pprof) list runtime.gopark
Total: 1699
ROUTINE ======================== runtime.gopark in /usr/local/go/src/runtime/proc.go
1691 1691 (flat, cum) 99.53% of Total
. . 361: mcall(park_m)
. . 362:}
. . 363:
. . 364:// Puts the current goroutine into a waiting state and unlocks the lock.
. . 365:// The goroutine can be made runnable again by calling goready(gp).
1691 1691 366:func goparkunlock(lock *mutex, reason waitReason, traceEv byte, traceskip int) {
. . 367: gopark(parkunlock_c, unsafe.Pointer(lock), reason, traceEv, traceskip)
. . 368:}
. . 369:
. . 370:func goready(gp *g, traceskip int) {
. . 371: systemstack(func() {
ROUTINE ======================== runtime.goparkunlock in /usr/local/go/src/runtime/proc.go
0 107 (flat, cum) 6.30% of Total
. . 367: gopark(parkunlock_c, unsafe.Pointer(lock), reason, traceEv, traceskip)
. . 368:}
. . 369:
. . 370:func goready(gp *g, traceskip int) {
. . 371: systemstack(func() {
. 107 372: ready(gp, traceskip, true)
. . 373: })
. . 374:}
. . 375:
. . 376://go:nosplit
. . 377:func acquireSudog() *sudog {
当然,pprof只能帮忙统计出go程序占用内存、cpu、goroutine等占用资源较多的函数,至于为什么该函数占用资源较高,就需要根据代码逻辑具体分析了。
点个赞
再走吧