基于containerd部署kubernetes v1.21.3

本次集群部署采用的容器技术是containerd

系统版本：CentOs8.1
k8s版本：v1.21.3
containerd版本：ctr containerd.io 1.4.3

master: 192.168.43.151
node1: 192.168.43.152
node2: 192.168.43.153

1、初始化配置
#下载CentOS 8的repo文件（用阿里的）

curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-8.repo
安装 kubelet kubeadm kubectl
master、node节点都需要执行

添加 kubernetes.repo 文件

cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

#关闭SElinux

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
systemctl enable --now kubelet  # 开机启动kubelet
systemctl disable --now firewalld  # 关闭防火墙

k8s要求关闭swap

swapoff -a && sysctl -w vm.swappiness=0  # 关闭swap
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab  # 取消开机挂载swap

2，安装containerd
安装前配置

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

设置必需的 sysctl 参数，这些参数在重新启动后仍然存在。

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
# Apply sysctl params without reboot

 sysctl --system

安装所需包

dnf install -y yum-utils device-mapper-persistent-data lvm2

新增 Docker 仓库

dnf config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

安装 containerd

dnf install -y containerd.io

配置 containerd

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

重启 containerd

systemctl restart containerd

3、安装kubelet kubeadm kubectl

dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

#–disableexcludes=kubernetes 禁掉除了这个之外的别的仓库

#kubelet配置修改

config kubelet cgroup

cat > /etc/sysconfig/kubelet <<EOF
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime=remote --container-runtime-endpoint=/run/containerd/containerd.sock
EOF

config CRI

cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

vi /etc/containerd/config.toml
# [plugins."io.containerd.grpc.v1.cri"] 下的 sandbox_image
# 将k8s.gcr.io/pause:3.2修改为如下
sandbox_image="registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2"

# 加上 SystemdCgroup = true
在[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]中加入
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

重启 containerd

sudo systemctl restart containerd

重启 kubelet

sudo systemctl restart kubelet

4，准备镜像
清空 ctr 中的所有镜像(因为之前我的版本是v1.20.4)

 ctr -n k8s.io images remove $(ctr -n k8s.io images list -q)

kubeadm config images list --kubernetes-version=1.21.3 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers -v 5
 # registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.21.3
 # registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.21.3
 # registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.21.3
 # registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.21.3
 # registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.4.1
 # registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.13-0
 # registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.0

手动下载镜像

#ctr -n k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
#ctr -n k8s.io images pull docker.io/coredns/coredns:1.8.0
$ ctr -n k8s.io images tag docker.io/coredns/coredns:1.8.0 registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.0   # 更改镜像TAG名称
 ### registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.0

# for i in $(kubeadm config images list --kubernetes-version=1.21.3 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers -v 5);do
  ctr -n k8s.io images pull ${i}
done

查看下载的镜像

 ctr -n k8s.io images ls | grep "google_containers"

5，启动集群
master启动

kubeadm init --kubernetes-version=v1.21.3 --apiserver-advertise-address=192.168.43.151 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers --service-cidr=10.10.0.0/16 --pod-network-cidr=192.168.0.0/16

–image-repository:指定镜像源

-----执行结果----

[root@ck8s1 ~]# kubeadm init --kubernetes-version=v1.21.3  --apiserver-advertise-address=192.168.43.151 --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers --service-cidr=10.10.0.0/16 --pod-network-cidr=192.168.0.0/16
[init] Using Kubernetes version: v1.21.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [ck8s1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.10.0.1 192.168.43.151]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [ck8s1 localhost] and IPs [192.168.43.151 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [ck8s1 localhost] and IPs [192.168.43.151 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 36.503764 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.21" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node ck8s1 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node ck8s1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: ne967i.qtyu5sa3hr3hyk8j
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.43.151:6443 --token ne967i.qtyu5sa3hr3hyk8j \
        --discovery-token-ca-cert-hash sha256:345b400454e3233020e9fc4e0ce4396447437f88ea317c839ffef490c5cce1a1 

---

#启动之后，需要再执行以下，否则在使用kubectl的时候会报8080端口错误：
非root用户

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

root用户

 export KUBECONFIG=/etc/kubernetes/admin.conf

6，应用calico网络

#kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
[root@ck8s1 ~]# kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy/calico-kube-controllers created

7，node节点加入：

dnf -y install kubectl kubelet kubeadm

kubeadm join 192.168.43.151:6443 --token ne967i.qtyu5sa3hr3hyk8j \
        --discovery-token-ca-cert-hash sha256:345b400454e3233020e9fc4e0ce4396447437f88ea317c839ffef490c5cce1a1 --cri-socket /run/containerd/containerd.sock

#前半部分是master输出的语句，最后添加cri的参数
–cri-socket /run/containerd/containerd.sock 的作用是指定cri为containerd

[root@ck8s2 ~]# kubeadm join 192.168.43.151:6443 --token ne967i.qtyu5sa3hr3hyk8j         --discovery-token-ca-cert-hash sha256:345b400454e3233020e9fc4e0ce4396447437f88ea317c839ffef490c5cce1a1 --cri-socket /run/containerd/containerd.sock
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

#查看node状态

#kubectl get node 
[root@ck8s1 ~]# kubectl get node  
NAME    STATUS   ROLES                  AGE     VERSION
ck8s1   Ready    control-plane,master   101m    v1.21.3
ck8s2   Ready    <none>                 11m     v1.21.3
ck8s3   Ready    <none>                 2m14s   v1.21.3

查看pod状态

kubectl get pod -A
[root@ck8s1 ~]# kubectl get pod -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-58497c65d5-g225d   1/1     Running   0          19m
kube-system   calico-node-2tkdh                          1/1     Running   0          2m49s
kube-system   calico-node-kwgvd                          1/1     Running   0          12m
kube-system   calico-node-lrv6z                          1/1     Running   0          19m
kube-system   coredns-6f6b8cc4f6-cgtdv                   1/1     Running   0          101m
kube-system   coredns-6f6b8cc4f6-thq5k                   1/1     Running   0          101m
kube-system   etcd-ck8s1                                 1/1     Running   0          101m
kube-system   kube-apiserver-ck8s1                       1/1     Running   0          101m
kube-system   kube-controller-manager-ck8s1              1/1     Running   0          101m
kube-system   kube-proxy-2zwv7                           1/1     Running   0          2m49s
kube-system   kube-proxy-5zgr8                           1/1     Running   0          12m
kube-system   kube-proxy-nbm49                           1/1     Running   0          101m
kube-system   kube-scheduler-ck8s1                       1/1     Running   0          101m

8，错误排查和解决方法
–报错1
[preflight] Running pre-flight checks
[WARNING FileExisting-tc]: tc not found in system path
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileContent–proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher
解决方法：

dnf install tc -y
modprobe br_netfilter && echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
–报错2：
Warning FailedCreatePodSandBox 53m (x9 over 73m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “k8s.gcr.io/pause:3.2”: failed to pull image “k8s.gcr.io/pause:3.2”: failed to pull and unpack image “k8s.gcr.io/pause:3.2”: failed to resolve reference “k8s.gcr.io/pause:3.2”: failed to do request: Head https://k8s.gcr.io/v2/pause/manifests/3.2: dial tcp 64.233.187.82:443: i/o timeout
Warning FailedCreatePodSandBox 43m (x3 over 44m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “k8s.gcr.io/pause:3.2”: failed to pull image “k8s.gcr.io/pause:3.2”: failed to pull and unpack image “k8s.gcr.io/pause:3.2”: failed to resolve reference “k8s.gcr.io/pause:3.2”: failed to do request: Head https://k8s.gcr.io/v2/pause/manifests/3.2: dial tcp 74.125.204.82:443: i/o timeout

解决方法：

第一种：
vim /etc/containerd/config.toml
[plugins.“io.containerd.grpc.v1.cri”] 下的 sandbox_image
将k8s.gcr.io/pause:3.2修改为如下（替换为可下载的镜像地址）
sandbox_image=“registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2”

其实还有其它的方法也能规避掉此问题，不用改config.toml文件，ctr image pull 把镜像下来，指定namespace 例如：

第二种：

下载镜像

ctr -n k8s.io image pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2

将下载的镜像打上k8s.gcr.io的tag

ctr -n k8s.io image tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 k8s.gcr.io/pause:3.2