Error: failed to run Kubelet: unable to determine runtime API version: rpc error: code = Unavailable

1.背景

新节点加入集群，一直抛出

Error: failed to run Kubelet: unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix: missing address"

root@raspberrypi:~/k8s/k8s-1.24.0/kubernetes/server/bin# docker version
Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:14 2023
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:14 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

kubeadm v1.25.0
kubelet v1.25.0
kubectl v1.25.0

1. 解决思路

1.1. 为kubelet加上相关参数(失败)

主动修改该service，加上--container-runtime=remote --container-runtime-endpoint=/var/run/cri-dockerd.sock

root@raspberrypi:~/k8s/k8s-1.24.0/kubernetes/server/bin# cat /etc/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/home/
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/bin/kubelet --container-runtime=remote  --container-runtime-endpoint=/var/run/cri-dockerd.sock
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

但再次尝试，一直反复抛出container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized， 且kubeadm join等待超时失败，节点一直无法加到集群中

1.2. 修改kubeadm/kubectl/kubelet 版本

将kubelet/kubectl/kubeadm版本由v1.25.0 修改为 v1.24.0

wget https://storage.googleapis.com/kubernetes-release/release/v1.24.0/kubernetes-server-linux-arm64.tar.gz

tar -zxvf kubernetes-server-linux-arm64.tar.gz 
ln -s $(pwd)/kubernetes/server/bin/kubelet /usr/bin/kubelet
ln -s $(pwd)/kubernetes/server/bin/kubeadm /usr/bin/kubeadm
ln -s $(pwd)/kubernetes/server/bin/kubectl /usr/bin/kubectl


# 再次执行kubeadm join 命令，成功加到集群中
kubeadm join 192.168.31.66:6443 --token jjoos3.8qm41fg2thrtufrn --discovery-token-ca-cert-hash sha256:18974f602e744dacf7fad69faf0793d5d802264158bea959030811f58f6fc5e8  --cri-socket unix:///var/run/cri-dockerd.sock --v=10

遇到的问题:

systemctl status kubelet

(1) failed to create shim task: OCI runtime create failed: runc create failed: invalid slice name

注意我们docker使用的Cgroud Driver, 建议使用默认

root@raspberrypi:~# cat /etc/docker/daemon.json 
{
  "registry-mirrors": [
    "https://docker.mirrors.ustc.edu.cn"
  ]
}![在这里插入图片描述](https://img-blog.csdnimg.cn/de3ee4c577fd42fa80fabf45e1ac48a7.png)


root@raspberrypi:~# docker info | grep Cg
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
WARNING: No swap limit support

# 如果节点没有安装相关的网络插件，可以从其他节点的相同目录拷贝一份过来
root@raspberrypi:~# ls /opt/cni/bin/
bandwidth    calico       dummy        host-device  ipvlan       portmap      static       vrf          
bin/         calico-ipam  firewall     host-local   loopback     ptp          tuning       
bridge       dhcp         flannel      install      macvlan      sbr          vlan      

Reference

Upgrade from k8s v1.23 to v1.24 fails due to restarting kubelet.
network plugin is not ready & cni config uninitialized
network plugin is not ready: cni config uninitialized