以kubelet为例使用go tool pprof分析服务性能

1

pprof是什么

pprof 是 go 语言自带的基础库，可以分析程序的运行情况，并且提供可视化的功能。它包含两个相关的库：runtime/pprof 和 net/http/pprof ， net/http/pprof 可以提供一个 HTTP 接口，获得性能数据，它的底层使用的 runtime/pprof。

2

pprof如何使用

一般如果我们使用 net/http/pprof 并开启了端口监听，如8001端口，我们就可以通过浏览器访问该服务pprof的页面，例如 http://127.0.0.1:8080/debug/pprof/ 

页面的下方还有各个命令的解释

我们常用的参数为

allocs	所有过去内存分配的采样
goroutine	所有当前goroutine的堆栈跟踪
heap	对活动对象的内存分配进行采样
profile	cpu profile采样

3

使用 pprof 分析 kubelet 性能

想要获取 kubelet 的 pprof 数据，首先需要使用kubectl proxy命令启动API server代理

# kubectl proxy --address='0.0.0.0'  --accept-hosts='^*$'
Starting to serve on [::]:8001

‍

直接使用 http://127.0.0.1:8080/debug/pprof/ 查看的是 apiserver 的性能指标，查看某个节点的 kubelet 的 url 为：

http://127.0.0.1:8001/api/v1/nodes/${NODENAME}/proxy/debug/pprof/

我们可以通过go tool pprof命令查看heap堆内存分配情况

go tool pprof http://127.0.0.1:8001/api/v1/nodes/192-168-249-10/proxy/debug/pprof/heap
Fetching profile over HTTP from http://127.0.0.1:8001/api/v1/nodes/192-168-249-10/proxy/debug/pprof/heap
Saved profile in /root/pprof/pprof.kubelet.alloc_objects.alloc_space.inuse_objects.inuse_space.004.pb.gz
File: kubelet
Build ID: 8cbf93d0e67344c230e43c23a345eb609612e432
Type: inuse_space
Time: Oct 19, 2022 at 5:11pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)

这时就进入了pprof的交互模式，我们输入top可以查看内存使用量前十的函数，如下所示

(pprof) top
Showing nodes accounting for 7780.25kB, 52.04% of 14950.70kB total
Showing top 10 nodes out of 191
      flat  flat%   sum%        cum   cum%
    1539kB 10.29% 10.29%     1539kB 10.29%  k8s.io/kubernetes/vendor/github.com/google/cadvisor/container/libcontainer.newContainerStats
 1032.02kB  6.90% 17.20%  1032.02kB  6.90%  k8s.io/kubernetes/vendor/github.com/euank/go-kmsg-parser/kmsgparser.(*parser).Parse.func1
 1025.78kB  6.86% 24.06%  1025.78kB  6.86%  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypeWithName
 1024.62kB  6.85% 30.91%  1024.62kB  6.85%  regexp/syntax.(*compiler).inst
  561.50kB  3.76% 34.67%   561.50kB  3.76%  k8s.io/kubernetes/vendor/golang.org/x/net/html.init
  528.17kB  3.53% 38.20%   528.17kB  3.53%  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/watch.(*Broadcaster).Watch.func1
  522.06kB  3.49% 41.69%   522.06kB  3.49%  k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/filedesc.(*File).initDecls
  516.76kB  3.46% 45.15%   516.76kB  3.46%  k8s.io/kubernetes/vendor/sigs.k8s.io/json/internal/golang/encoding/json.typeFields
  516.64kB  3.46% 48.60%   516.64kB  3.46%  io.ReadAll
  513.69kB  3.44% 52.04%   513.69kB  3.44%  k8s.io/kubernetes/vendor/google.golang.org/protobuf/internal/strs.(*Builder).AppendFullName

可见libcontainer.newContainerStats函数占用内存最多，占用了10%的内存。如果是某个进程占用内存很高，就可以以此为依据，排查占用内存最多的几个函数，再根据源码分析具体原因。

刚才分析的是 heap 内存情况，同样可以根据需要分析前面说的所有采样。

#所有过去内存分配的采样
go tool pprof http://127.0.0.1:8001/debug/pprof/allocs
#对活动对象的内存分配进行采样（活动）
go tool pprof http://127.0.0.1:8001/debug/pprof/heap
# 下载 cpu profile，默认从当前开始收集 30s 的 cpu 使用情况，需要等待 30s
go tool pprof http://127.0.0.1:8001/debug/pprof/profile
# wait 120s
go tool pprof http://127.0.0.1:8001/debug/pprof/profile?seconds=120    
#导致同步原语阻塞的堆栈跟踪
go tool pprof http://127.0.0.1:8001/debug/pprof/block
#所有当前goroutine的堆栈跟踪
go tool pprof http://127.0.0.1:8001/debug/pprof/goroutine
#争用互斥锁持有者的堆栈跟踪
go tool pprof http://127.0.0.1:8001/debug/pprof/mutex
#当前程序的执行轨迹。
go tool pprof http://127.0.0.1:8001/debug/pprof/trace

例如我们查看 apiserver 的 goroutine

# go tool pprof http://127.0.0.1:8001/debug/pprof/goroutine
Fetching profile over HTTP from http://127.0.0.1:8001/debug/pprof/goroutine
Saved profile in /root/pprof/pprof.kube-apiserver.goroutine.001.pb.gz
File: kube-apiserver
Type: goroutine
Time: Oct 19, 2022 at 5:27pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 1691, 99.53% of 1699 total
Dropped 128 nodes (cum <= 8)
Showing top 10 nodes out of 116
      flat  flat%   sum%        cum   cum%
      1691 99.53% 99.53%       1691 99.53%  runtime.gopark
         0     0% 99.53%         67  3.94%  bufio.(*Reader).Read
         0     0% 99.53%         78  4.59%  bytes.(*Buffer).ReadFrom
         0     0% 99.53%         59  3.47%  context.propagateCancel.func1
         0     0% 99.53%         78  4.59%  crypto/tls.(*Conn).Read
         0     0% 99.53%         78  4.59%  crypto/tls.(*Conn).readFromUntil
         0     0% 99.53%         78  4.59%  crypto/tls.(*Conn).readRecord (inline)
         0     0% 99.53%         78  4.59%  crypto/tls.(*Conn).readRecordOrCCS
         0     0% 99.53%         78  4.59%  crypto/tls.(*atLeastReader).Read
         0     0% 99.53%         78  4.59%  internal/poll.(*FD).Read

pprof同时还能够通过 list 命令查看指定函数的具体函数代码

(pprof) list runtime.gopark
Total: 1699
ROUTINE ======================== runtime.gopark in /usr/local/go/src/runtime/proc.go
      1691       1691 (flat, cum) 99.53% of Total
         .          .    361:  mcall(park_m)
         .          .    362:}
         .          .    363:
         .          .    364:// Puts the current goroutine into a waiting state and unlocks the lock.
         .          .    365:// The goroutine can be made runnable again by calling goready(gp).
      1691       1691    366:func goparkunlock(lock *mutex, reason waitReason, traceEv byte, traceskip int) {
         .          .    367:  gopark(parkunlock_c, unsafe.Pointer(lock), reason, traceEv, traceskip)
         .          .    368:}
         .          .    369:
         .          .    370:func goready(gp *g, traceskip int) {
         .          .    371:  systemstack(func() {
ROUTINE ======================== runtime.goparkunlock in /usr/local/go/src/runtime/proc.go
         0        107 (flat, cum)  6.30% of Total
         .          .    367:  gopark(parkunlock_c, unsafe.Pointer(lock), reason, traceEv, traceskip)
         .          .    368:}
         .          .    369:
         .          .    370:func goready(gp *g, traceskip int) {
         .          .    371:  systemstack(func() {
         .        107    372:    ready(gp, traceskip, true)
         .          .    373:  })
         .          .    374:}
         .          .    375:
         .          .    376://go:nosplit
         .          .    377:func acquireSudog() *sudog {

当然，pprof只能帮忙统计出go程序占用内存、cpu、goroutine等占用资源较多的函数，至于为什么该函数占用资源较高，就需要根据代码逻辑具体分析了。

点个赞

再走吧