基于ubuntu20.04安装kubernetes1.27.1(使用cri-docker)

1.环境准备

192.168.1.60 master
192.168.1.61 node1
192.168.1.62 node2
192.168.1.63 node3

1.1配置hostname，host文件

不修改hostname会导致主机名相同，安装网络创建后，不同node节点的pod通信会有问题

1.2 配置内核转发及网桥过滤

cat /etc/sysctl.d/k8s.conf 
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
sysctl -p  /etc/sysctl.d/k8s.conf 

2.安装docker

2.1使用阿里源安装docker

sudo apt-get update
sudo apt-get -y install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://mirrors.aliyun.com/docker-ce/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get -y update
apt install docker 

2.2配置docker使用systemd

 vim /etc/docker/daemon.json 
{
  "exec-opts":["native.cgroupdriver=systemd"]

}

2.3启动docker

systemctl daemon-reload  &&  systemctl restart docker  && systemctl enable docker 

3.安装cri-docker

3.1.二进制安装

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.5/cri-dockerd-0.2.5.amd64.tgz

tar -xf cri-dockerd-0.2.5.amd64.tgz 
mv cri-dockerd/cri-dockerd  /usr/local/bin/
 scp /usr/local/bin/cri-dockerd  192.168.1.62:/usr/local/bin/
 scp /usr/local/bin/cri-dockerd  192.168.1.63:/usr/local/bin/
 scp /usr/local/bin/cri-dockerd  192.168.1.60:/usr/local/bin/

配置service和socker文件

cat  /etc/systemd/system/cri-docker.service 
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket

[Service]
Type=notify
ExecStart=/usr/local/bin/cri-dockerd --network-plugin=cni   --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target
cat /lib/systemd/system/cri-docker.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service

[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

启动

systemctl daemon-reload   && systemctl enable cri-docker   && systemctl start cri-docker && systemctl enable --now cri-docker.socket 

3.2直接使用dpkg 安装 cri-docker

dpkg 安装方便，不易出错

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.1/cri-dockerd_0.3.1.3-0.ubuntu-focal_amd64.deb
dpkg -i cri-dockerd_0.3.1.3-0.ubuntu-focal_amd64.deb
systemctl start cri-docker

修改配置文件pause镜像使用过阿里源，默认使用的是国外的，下载不了pause

vi /lib/systemd/system/cri-docker.service
ExecStart=/usr/bin/cri-dockerd --network-plugin=cni   --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9

systemctl daemon-reload && systemctl restart cri-docker.service
查看状态
systemctl status cri-docker.service

4.安装 kubeadm kubelet kubectl

关闭交换分区
swapoff

apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl

查看版本

root@master:~# kubeadm  version
kubeadm version: &version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:20:04Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}

5.准备镜像

查看所需要哪些镜像，然后修改为阿里云的镜像后下载
5.1查看kubernetes 1.27.1所需要的镜像

root@master:~#  kubeadm  config images list --kubernetes-version v1.27.1
W0511 20:12:21.307628   84645 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.1, falling back to the nearest etcd version (3.5.7-0)
registry.k8s.io/kube-apiserver:v1.27.1
registry.k8s.io/kube-controller-manager:v1.27.1
registry.k8s.io/kube-scheduler:v1.27.1
registry.k8s.io/kube-proxy:v1.27.1
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.7-0
registry.k8s.io/coredns/coredns:v1.10.1

修改为阿里云镜像，运行镜像脚本并运行下载

[root@k8s-master01 ~]# cat images.sh 
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.27.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.27.1
docker pull  registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.27.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1
docker pull  registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
docker pull  registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.7-0
docker pull  registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1
[root@k8s-master01 ~]# sh images.sh 

6.初始化master

添加–cri-socket=unix:///run/cri-dockerd.sock，指定为cri-docker

kubeadm init   --apiserver-advertise-address=192.168.1.50  --apiserver-bind-port=6443 --kubernetes-version=1.27.1  --pod-network-cidr=10.200.0.0/16 --service-cidr=192.168.3.0/24 --service-dns-domain=cluster.local  --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --ignore-preflight-errors=swap   --cri-socket=unix:///run/cri-dockerd.sock

初始化成功

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.60:6443 --token hvzgvp.x3hlo9qac22abuab \
	--discovery-token-ca-cert-hash sha256:353f5aea8ca0aa10e6da69a4aaa37da58d63db7d1b133784d1bcabc9bba8c860   --cri-socket=unix:///run/cri-dockerd.sock 

master 执行
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id-u):$(id -g) $HOME/.kube/config

7.node节点加入

kubeadm join 192.168.1.60:6443 --token hvzgvp.x3hlo9qac22abuab
–discovery-token-ca-cert-hash sha256:353f5aea8ca0aa10e6da69a4aaa37da58d63db7d1b133784d1bcabc9bba8c860 --cri-socket=unix:///run/cri-dockerd.sock

将master 节点的config文件拷贝到node节点，node节点也可以使用kubectl get 命令
scp .kube/config 192.168.1.92

8.遇到的问题

master 初始化报错

`root@server:~# kubeadm init   --apiserver-advertise-address=192.168.1.80  --apiserver-bind-port=6443 --kubernetes-version=1.24.1  --pod-network-cidr=10.222.0.0/16 --service-cidr=192.168.6.0/24 --service-dns-domain=cluster.local  --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --ignore-preflight-errors=swap    --cri-socket=unix:///run/cri-dockerd.sock
[init] Using Kubernetes version: v1.24.1
[preflight] Running pre-flight checks
	[WARNING SystemVerification]: missing optional cgroups: blkio
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR CRI]: container runtime is not running: output: time="2023-06-26T15:14:01Z" level=fatal msg="validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///run/cri-dockerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

master初始化失败，cri-docker和kubelet等版本不匹配的问题.安装cri-docker最新版可以后可以初始化，
或者cri-docker和kubeadm 均安装最新版

2.The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled
tail -f syslog 查看日志
May 11 18:44:38 ubuntu20 cri-dockerd[27458]: time=“2023-05-11T18:44:38+08:00” level=info msg=“Pulling the image without credentials. Image: registry.k8s.io/pause:3.6”

默认配置是国外的镜像拉取不到镜像，配置国内镜像后，重新初始化

 kubeadm reset --cri-socket=unix:///run/cri-dockerd.sock #重置
kubeadm init   --apiserver-advertise-address=192.168.1.60  --apiserver-bind-port=6443 --kubernetes-version=1.27.1  --pod-network-cidr=10.201.0.0/16 --service-cidr=192.168.4.0/24 --service-dns-domain=cluster.local  --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --ignore-preflight-errors=swap   --cri-socket=unix:///run/cri-dockerd.sock 

9.配置网络插件

1.方法一

wget https://get.helm.sh/helm-v3.9.0-linux-amd.tar.gz
wget https://get.helm.sh/helm-v3.9.0-linux-amd.tar.gz
tar -xf helm-v3.9.0-linux-amd.tar.gz
mv helm  /usr/local/bin/
root@master:~/linux-amd64# helm repo add hybridnet https://alibaba.github.io/hybridnet/
"hybridnet" has been added to your repositories

更新
root@master:~/linux-amd64# helm repo update

配置overlay pod网络
root@master:~/linux-amd64# helm install hybridnet hybridnet/hybridnet -n kube-system --set init.cidr=10.201.0.0/16
W0511 21:35:23.614353  114776 warnings.go:70] spec.template.spec.nodeSelector[beta.kubernetes.io/os]: deprecated since v1.14; use "kubernetes.io/os" instead
W0511 21:35:23.614408  114776 warnings.go:70] spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead
NAME: hybridnet
LAST DEPLOYED: Thu May 11 21:35:22 2023
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1

给node打标签，打完标签，pod才会运行

root@master:~/linux-amd64# kubectl label node node1 node-role.kubernetes.io/master=
node/node1 labeled
root@master:~/linux-amd64# kubectl label node node2 node-role.kubernetes.io/master=
node/node2 labeled
root@master:~/linux-amd64# kubectl label node node3 node-role.kubernetes.io/master=
node/node3 labeled

2. 方法二：由于下载镜像慢，可以从github 下载完整的包，导入镜像后，运行calico.yaml 即可

https://github.com/projectcalico/calico/releases/download/v3.25.1/release-v3.25.1.tgz
tar -xf release-v3.25.1.tgz
docker load -i release-v3.25.1/images/*
kubectl -f /root/release-v3.25.1/manifests/calico.yaml
3.24.5 calico yaml

calico.yaml