kubeadm安装kubernetes集群_reason networkpluginnotready message network plugi(2)

既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，涵盖了95%以上Go语言开发知识点，真正体系化！

由于文件比较多，这里只是将部分目录截图出来，全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频，并且后续会持续更新

如果你需要这些资料，可以戳这里获取

下载cri-dockerd

下载官网发布的版本

https://github.com/Mirantis/cri-dockerd/releases/tag/v0.2.5

自己clone源代码编译

cd cri-dockerd
mkdir bin
go build -o bin/cri-dockerd
mkdir -p /usr/local/bin
install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
cp -a packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable cri-docker.service
systemctl enable --now cri-docker.socket

cri-dockerd服务配置

1. 创建/etc/systemd/system/cri-docker.socket

[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service
[Socket]
ListenStream=%t/cri-dockerd.sock # 此处为sock文件的存放路径
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target

2. 创建/etc/systemd/system/cri-docker.service

[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket
[Service]
Type=notify
# 此处为启动的命令行
# 注意启动的dockerd的路径
# 注意网络插件及pause的配置
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --networkplugin=cni \
--pod-infra-containerimage=registry.aliyuncs.com/google_containers/pause:3.7  
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target

3. 启动并验证服务

# 重新加载配置
systemctl daemon-reload
# 设置为开机自启动
systemctl enable cri-docker
# 启动服务
systemctl enable --now cri-docker
# 检查服务状态
systemctl status cri-docker

containerd

二进制文件安装

安装containerd

从https://github.com/containerd/containerd/releases下载containerd-<VERSION>-<OS>-<ARCH>.tar.gz

$ cd /usr/local
$ tar Cxzvf /usr/local containerd-1.6.2-linux-amd64.tar.gz
bin/
bin/containerd-shim-runc-v2
bin/containerd-shim
bin/ctr
bin/containerd-shim-runc-v1
bin/containerd
bin/containerd-stress

配置/usr/lib/systemd/system/containerd.service

# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
#uncomment to enable the experimental sbservice (sandboxed) version of containerd/cri integration
#Environment="ENABLE_CRI_SANDBOXES=sandboxed"
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

启动服务

systemctl daemon-reload
systemctl enable --now containerd

安装 runc

从https://github.com/opencontainers/runc/releases下载runc.<ARCH>二进制文件

cd /usr/local/sbin/
mv runc.amd64 runc
chmod 755 runc

安装cni插件

从https://github.com/containernetworking/plugins/releases下载cni-plugins-<OS>-<ARCH>-<VERSION>.tgz存档，然后在下面解压：/opt/cni/bin

$ mkdir -p /opt/cni/bin
$ tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.1.1.tgz
./
./macvlan
./static
./vlan
./portmap
./host-local
./vrf
./bridge
./tuning
./firewall
./host-device
./sbr
./loopback
./dhcp
./ptp
./ipvlan
./bandwidth

修改containerd配置

 mkdir /etc/containerd
# 生成配置文件
containerd config default > /etc/containerd/config.toml
# 重载沙箱（pause）镜像 
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.7"
# 设置cgroup驱动
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

# 重启containerd
sudo systemctl restart containerd

circtl默认链接unix:///var/run/dockershim.sock，所以需要修改circtl配置文件

cat <<EOF> /etc/crictl.yaml 
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

拉取镜像验证

crictl pull nginx:1.20.2
crictl images ls

kubeadm

1. 安装相关软件

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

2. 下载gpg密钥：这里使用阿里云的

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg \
https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg

3. 设置kubernetes镜像源

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg]  \
https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" | sudo tee \
/etc/apt/sources.list.d/kubernetes.list

4. 更新apt软件索引，并查看相关软件的可用版本

sudo apt-get update
apt-cache madison kubelet kubeadm kubectl

5. 安装特定版本

sudo apt-get install -y kubelet=<VERSION_STRING> kubeadm=<VERSION_STRING>
kubectl=<VERSION_STRING>
例如：
sudo apt-get install -y kubelet=1.24.1-00 kubeadm=1.24.1-00 kubectl=1.24.1-00

6. 检查

# kubeadm
kubeadm version
# kubectl
kubectl version
# kubelet
systemctl status kubelet

注意：kubelet在刚安装完成时，会处于一个自动启动状态，每10s启动一次，在没有完成初始化之前它
一致处于这种状态，所以不要纠结于kubelet安装之后没有启动。

初始化配置（仅master节点）

生成默认配置文件

kubeadm config print init-defaults > init.default.yaml

修改配置文件

# 修改地址 节点IP地址
localAPIEndpoint.advertiseAddress: 192.168.56.101
# 修改套接字，如果使用cri-docker需要修改
nodeRegistration.criSocket: unix:///var/run/cri-dockerd.sock
# 修改节点名称
nodeRegistration.name: master1
# 修改镜像仓库地址为国内开源镜像库
imageRepository: registry.aliyuncs.com/google_containers
# 修改版本号
kubernetesVersion: 1.24.1
# 增加podSubnet，由于后续会安装flannel 网络插件，该插件必须在集群初始化时指定pod地址
# 10.244.0.0/16 为flannel组件podSubnet默认值，集群配置与网络组件中的配置需保持一致
podSubnet: 10.244.0.0/16

拉取相关镜像

sudo kubeadm config images pull --config=init.default.yaml

初始化集群

# 通过配置文件初始化
sudo kubeadm init --config=init.default.yaml
# 通过参数初始化
kubeadm init --image-repository registry.aliyuncs.com/google_containers --
kubernetes-version=v1.24.1 --pod-network-cidr=10.244.0.0/16 --apiserveradvertise-address=192.168.239.142 --cri-socket unix:///var/run/cri-dockerd.sock

若当前用户为普通用户，请执行以下命令

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

当前用户为root用户，请配置环境变量：

# /etc/profile 末尾添加环境变量
export KUBECONFIG=/etc/kubernetes/admin.conf
# 执行命令，立即生效
source /etc/profile

查看节点状态

# kubectl get node
NAME      STATUS     ROLES           AGE   VERSION
master1   NotReady   control-plane   89s   v1.24.1

可以看到节点状态为NotReady

查看kubelet状态

# systemctl status kubelet
  Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Sun 2022-09-04 17:18:25 CST; 5min ago
       Docs: https://kubernetes.io/docs/home/
   Main PID: 2146 (kubelet)
      Tasks: 16 (limit: 2236)
     Memory: 34.8M
        CPU: 14.635s
     CGroup: /system.slice/kubelet.service
             └─2146 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=unix:>

Sep 04 17:23:05 master1 kubelet[2146]: E0904 17:23:05.440384    2146 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not i>
Sep 04 17:23:10 master1 kubelet[2146]: E0904 17:23:10.441932    2146 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not i>
Sep 04 17:23:15 master1 kubelet[2146]: E0904 17:23:15.443737    2146 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not i>
Sep 04 17:23:20 master1 kubelet[2146]: E0904 17:23:20.445438    2146 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not i>
Sep 04 17:23:25 master1 kubelet[2146]: E0904 17:23:25.447628    2146 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not i>
Sep 04 17:23:30 master1 kubelet[2146]: E0904 17:23:30.451519    2146 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not i>
Sep 04 17:23:35 master1 kubelet[2146]: E0904 17:23:35.454570    2146 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not i>
Sep 04 17:23:40 master1 kubelet[2146]: E0904 17:23:40.459534    2146 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not i>
Sep 04 17:23:45 master1 kubelet[2146]: E0904 17:23:45.465543    2146 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not i>
Sep 04 17:23:50 master1 kubelet[2146]: E0904 17:23:50.468645    2146 kubelet.go:2344] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not i>
~

可以看到网络插件为准备好

使用命令查看所有pod可以看到有coredns为准备好

kubectl get pod --all-namespaces
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   coredns-74586cf9b6-22qc5          0/1     Pending   0          6m3s
kube-system   coredns-74586cf9b6-qx9ql          0/1     Pending   0          6m3s
kube-system   etcd-master1                      1/1     Running   0          6m7s
kube-system   kube-apiserver-master1            1/1     Running   0          6m7s
kube-system   kube-controller-manager-master1   1/1     Running   0          6m7s
kube-system   kube-proxy-dgmcn                  1/1     Running   0          6m3s
kube-system   kube-scheduler-master1            1/1     Running   0          6m7s

安装网络插件

Kubernetes 定义了 CNI 标准，有很多网络插件，这里我选择最常用的 Flannel，可以在它的 GitHub 仓库里（https://github.com/flannel-io/flannel/）找到相关文档。它安装也很简单，只需要使用项目的“kube-flannel.yml”在 Kubernetes 里部署一下就好了。

下载地址

你可以使用curl下载下来

curl https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml >> flannel.yml

如果前面有设置podSubnet那么，你需要修改文件里的“net-conf.json”字段，把 Network 改成刚才 kubeadm 的参数 --pod-network-cidr 设置的地址段，例如：

  net-conf.json: |
    {
      "Network": "10.10.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

kubectl apply -f flannel.yml

然后再查看节点状态就可以看到master节点已经ready了

# kubectl get node
NAME      STATUS   ROLES           AGE   VERSION
master1   Ready    control-plane   21m   v1.24.1

查看所有pod节点也都是正常，这里需要注意kube-flannel可以能初始化会有所延迟。

# kubectl get pod --all-namespaces
NAMESPACE      NAME                              READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-8x697             1/1     Running   0          2m12s
kube-system    coredns-74586cf9b6-4vqhq          1/1     Running   0          9m26s
kube-system    coredns-74586cf9b6-6s6mk          1/1     Running   0          9m26s
kube-system    etcd-master1                      1/1     Running   1          9m40s
kube-system    kube-apiserver-master1            1/1     Running   1          9m40s
kube-system    kube-controller-manager-master1   1/1     Running   0          9m40s
kube-system    kube-flannel-ds-thzgs             1/1     Running   0          5m59s
kube-system    kube-proxy-8f28v                  1/1     Running   0          9m26s
kube-system    kube-scheduler-master1            1/1     Running   1          9m40s

开启kube-proxy的ipvs模式

# 修改mod
kubectl edit cm kube-proxy -n kube-system
修改：mode: "ipvs"
# 删除现有kube-proxy pod
kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'

节点

在master节点获取加入节点命令

sudo kubeadm token create --print-join-command

注意：如果容器运行时不是contained需要在命令后面配置sock的url，例如cri-dockerd就需要配置--cri-socket unix:///var/run/cri-dockerd.sock

重置节点

如果节点不需要或者出错需要删除时我们需要重置节点，重置步骤为

既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，涵盖了95%以上Go语言开发知识点，真正体系化！

由于文件比较多，这里只是将部分目录截图出来，全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频，并且后续会持续更新

如果你需要这些资料，可以戳这里获取

delete pod “$1” -n kube-system")}’

## 节点


在master节点获取加入节点命令

sudo kubeadm token create --print-join-command

注意：如果容器运行时不是contained需要在命令后面配置sock的url，例如cri-dockerd就需要配置`--cri-socket unix:///var/run/cri-dockerd.sock`


### 重置节点


如果节点不需要或者出错需要删除时我们需要重置节点，重置步骤为




[外链图片转存中...(img-tzfcJJ2s-1715904839974)]
[外链图片转存中...(img-9CaaYobQ-1715904839975)]
[外链图片转存中...(img-0EVjv7q8-1715904839975)]

**既有适合小白学习的零基础资料，也有适合3年以上经验的小伙伴深入学习提升的进阶课程，涵盖了95%以上Go语言开发知识点，真正体系化！**

**由于文件比较多，这里只是将部分目录截图出来，全套包含大厂面经、学习笔记、源码讲义、实战项目、大纲路线、讲解视频，并且后续会持续更新**

**[如果你需要这些资料，可以戳这里获取](https://bbs.csdn.net/topics/618658159)**