k8s源码debug

概述

本文将 linux k8s 源码编译及单集群测试 基础上，对k8s源码如何调试做详细的说明。

启动本地集群

如果需要，用 make clean 清除未编译的执行程序。通过 hack/lock-up-cluster.sh 脚本启动本地集群

postman 请求

证书及token

# 查看当前 k8s 是否有 service account
cluster/kubectl.sh get sa
cluster/kubectl.sh get secret
cluster/kubectl.sh create sa forpostman
cluster/kubectl.sh describe sa forpostman


curl https://172.16.215.128:6443/apis
curl https://localhost:6443/apis
curl http://localhost:6443/apis

# 查看当前 k8s 是否有 service account
[root@test kubernetes]# cluster/kubectl.sh get sa
NAME      SECRETS   AGE
default   0         25m
# 也是没有任何 secret
[root@test kubernetes]# cluster/kubectl.sh get secret
No resources found in default namespace.

# 创建 service account
[root@test kubernetes]# cluster/kubectl.sh create sa forpostman
serviceaccount/forpostman created
[root@test kubernetes]# cluster/kubectl.sh get sa
NAME         SECRETS   AGE
default      0         29m
forpostman   0         8s
# 查看 forpostman 的详细信息
[root@test kubernetes]# cluster/kubectl.sh describe sa forpostman
Name:                forpostman
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>
[root@test kubernetes]# 

postman-sa-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: postman-sa-secret
  annotations:
    kubernetes.io/service-account.name: forpostman
type: kubernetes.io/service-account-token     

创建

[root@test kubernetes]# cluster/kubectl.sh apply -f ~/k8s_ymal/postman-sa-secret.yaml
secret/postman-sa-secret created
[root@test kubernetes]# cluster/kubectl.sh get secret
NAME                TYPE                                  DATA   AGE
postman-sa-secret   kubernetes.io/service-account-token   3      68s
[root@test kubernetes]# cluster/kubectl.sh describe sa forpostman
Name:                forpostman
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              postman-sa-secret
Events:              <none>

# 获取 token
[root@test kubernetes]# cluster/kubectl.sh describe secret  postman-sa-secret
Name:         postman-sa-secret
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: forpostman
              kubernetes.io/service-account.uid: 4ae0543b-60ef-4e71-b92b-01a1254d42e9

Type:  kubernetes.io/service-account-token

Data
====
namespace:  7 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6ImtpQ29BN3NUVGZvWnF2MnZ1dXZ6YjhTLUNnYmtTcWlJU1NqX25pVUFnV28ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InBvc3RtYW4tc2Etc2VjcmV0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImZvcnBvc3RtYW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI0YWUwNTQzYi02MGVmLTRlNzEtYjkyYi0wMWExMjU0ZDQyZTkiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpmb3Jwb3N0bWFuIn0.aSSaghkj-QeJzNZV_qFjLzh7YrFJAwj03kNLBWTckG3VbVgd79zIiTiVOOeTB7nwPbTx-l11a4ZoSskrpkAJdkY_UjOvx1mltwV0a8m--b5W8BrP4hDfwvg8RzngiSoyfo0uKMuDL6u1R37sS1Txq0uQoU-LaWrq85Q7UBqC1AUdQECdEDUMEUr5cu6uM1fxxxq6aDJ59jMV1DNYsu6F71y3226K-PQEmIugDLER16F-1dr8cXefaHaCuZoX__twec04XVuSYyn2DJ8H6s1zKWWTE1s9yw9aavW6H0_gnvW-8uQywv2yQxWRiVu4nCir8OCXpaH_0mc1FiBTi7Gn2Q
ca.crt:     1289 bytes
[root@test kubernetes]# 

查看系统有哪些权限，上面的 sa 账户还没有授权

[root@test kubernetes]# cluster/kubectl.sh get clusterrole
NAME                                                                   CREATED AT
admin                                                                  2024-02-05T07:08:10Z
cluster-admin                                                          2024-02-05T07:08:10Z
edit                                                                   2024-02-05T07:08:10Z
system:aggregate-to-admin                                              2024-02-05T07:08:10Z
system:aggregate-to-edit                                               2024-02-05T07:08:10Z
system:aggregate-to-view                                               2024-02-05T07:08:10Z

授权

# 给 rolebinding 起个名字 forpostmanadmin
cluster/kubectl.sh create rolebinding forpostmanadmin --clusterrole cluster-admin --serviceaccount default:forpostman


[root@test kubernetes]# cluster/kubectl.sh create rolebinding forpostmanadmin --clusterrole cluster-admin --serviceaccount default:forpostman
rolebinding.rbac.authorization.k8s.io/forpostmanadmin created
# 创建成功
[root@test kubernetes]# cluster/kubectl.sh get rolebinding
NAME              ROLE                        AGE
forpostmanadmin   ClusterRole/cluster-admin   50s

导出证书

# 1.先导至集群本地
[root@test kubernetes]# cluster/kubectl.sh get secret  postman-sa-secret -o jsonpath="{.data['ca\.crt']}" | base64 -d > /tmp/ca.crt
[root@test kubernetes]# ls -l /tmp/ca.crt
-rw-r--r--. 1 root root 1289 2月   5 16:49 /tmp/ca.crt
[root@test kubernetes]# 


# 2.下载 ca.crt

安装go-delve

centos端

[root@test ~]# go install github.com/go-delve/delve/cmd/dlv@latest
go: downloading github.com/go-delve/delve v1.22.0
go: downloading github.com/spf13/cobra v1.7.0
go: downloading github.com/sirupsen/logrus v1.9.3
go: downloading github.com/mattn/go-isatty v0.0.20
....
go: downloading github.com/russross/blackfriday/v2 v2.1.0
go: downloading golang.org/x/exp v0.0.0-20230224173230-c95f2b4c22f2
go: downloading github.com/rivo/uniseg v0.2.0
[root@test ~]# dlv
Delve is a source level debugger for Go programs.

Delve enables you to interact with your program by controlling the execution of the process,
evaluating variables, and providing information of thread / goroutine state, CPU register state and more.

The goal of this tool is to provide a simple yet powerful interface for debugging Go programs.

Pass flags to the program you are debugging using `--`, for example:

`dlv exec ./hello -- server --config conf/config.toml`

Usage:
  dlv [command]

Available Commands:
  attach      Attach to running process and begin debugging.
  completion  Generate the autocompletion script for the specified shell
  connect     Connect to a headless debug server with a terminal client.
  core        Examine a core dump.
  dap         Starts a headless TCP server communicating via Debug Adaptor Protocol (DAP).
  debug       Compile and begin debugging main package in current directory, or the package specified.
  exec        Execute a precompiled binary, and begin a debug session.
  help        Help about any command
  test        Compile test binary and begin debugging program.
  trace       Compile and begin tracing program.
  version     Prints version.

Additional help topics:
  dlv backend    Help about the --backend flag.
  dlv log        Help about logging flags.
  dlv redirect   Help about file redirection.

Use "dlv [command] --help" for more information about a command.
[root@test ~]# 

go-delve

delve官方文档

k8s debug

启动本地集群从而 debug 以 api server 为例。

常用命令

ps aux | grep kube
ps aux | grep kube-apiserver

api server

修改参数

1. 修改 hack/lib/golang.sh文件，从而使得编译器不去优化掉支持 debug 的信息；
2. 禁止 -w -s ，保留文件名，行号；
3. 加上 -gcflags= “all=-N -l” ，禁止优化和肉联。

# ./hack/lib/golang.sh
[root@test kubernetes]# vim hack/lib/golang.sh 
# 当然也可是不修改，设置环境变量 DBG=1

# 我的配置如下
#if [[ "${DBG:-}" == 1 ]]; then
     # Debugging - disable optimizations and inlining.
  #   gogcflags="${gogcflags} -N -l"
 #fi
 gogcflags="all=-N -l"

 goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}"
 #if [[ "${DBG:-}" != 1 ]]; then
     # Not debugging - disable symbols and DWARF.
  #   goldflags="${goldflags} -s -w"
 #fi

重启 api-server

如果要调试 api server ，先关闭其进程，再以 dlv 启动

通过 delve 连接 debug server 并开始调试

ps aux | grep kube-apiserver
ps -a | grep kube*

使用dlv启动 api-server

dlv exec /data/soft/go/src/k8s.io/kubernetes/_output/bin/kube-apiserver --headless=true  --listen=:12345 --api-version=2  -- --authorization-mode=Node,RBAC  --cloud-provider= --cloud-config=   --v=3 --vmodule= --audit-policy-file=/tmp/kube-audit-policy-file --audit-log-path=/tmp/kube-apiserver-audit.log --authorization-webhook-config-file= --authentication-token-webhook-config-file= --cert-dir=/var/run/kubernetes --egress-selector-config-file=/tmp/kube_egress_selector_configuration.yaml --client-ca-file=/var/run/kubernetes/client-ca.crt --kubelet-client-certificate=/var/run/kubernetes/client-kube-apiserver.crt --kubelet-client-key=/var/run/kubernetes/client-kube-apiserver.key --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=true --service-account-issuer=https://kubernetes.default.svc --service-account-jwks-uri=https://kubernetes.default.svc/openid/v1/jwks --service-account-signing-key-file=/tmp/kube-serviceaccount.key --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction --disable-admission-plugins= --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving-kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=AllAlpha=false --external-hostname=localhost --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key 

本地调试

# 连接
[root@VM-16-2-centos ~]# dlv connect localhost:12345
Type 'help' for list of commands.

break cmd/kube-apiserver/apiserver.go:33

continue

执行结果如下图

visual studio code调试

使用 visual studio code 进行调试

gogcflags

结束

k8s源码debug ，至此结束，如有疑问，欢迎评论区留言。