k8s 1.28安装

容器运行时，containerd

按照官方的指导，需要安装runc和cni插件，提示的安装方式，有三种：

1. 二进制安装包
2. 源码
3. apt-get 或 dnf安装
   我们这里选用第三种，找到docker官方提供的安装方式

ubuntu-containerd

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# Add the repository to Apt sources:
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

这里我都是直接安装的最新版

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

你也可以通过指令查看可以安装的指定版本

apt-cache madison docker-ce| awk '{ print $3 }'

centos-containerd

添加源

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

安装最新版本

sudo yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

注意要生成containerd 配置文件，并修改SystemdCgroup为true

containerd config default > /etc/containerd/config.toml

设置容器运行时

crictl config runtime-endpoint unix:///run/containerd/containerd.sock

cni

前面的docker安装种保护了containerd.io 包，他包含了 runc 但是不包含 CNI plugins，因此需要补充CNI插件
你可以从找到cni插件的下载连接，官方，也可以使用我之前下载的

链接：https://pan.baidu.com/s/1eHV4KuM_1bUTuZ_2zW3UQg?pwd=ypdi 
提取码：ypdi

将文件cni-plugins-linux-arm64-v1.3.0.tgz拷贝至节点

$ mkdir -p /opt/cni/bin
$ tar Cxzvf /opt/cni/bin cni-plugins-linux-arm64-v1.3.0.tgz
./
./macvlan
./static
./vlan
./portmap
./host-local
./vrf
./bridge
./tuning
./firewall
./host-device
./sbr
./loopback
./dhcp
./ptp
./ipvlan
./bandwidth

kubelet

前置条件

关闭交换分区 swappoff

#关闭分区
swapoff -a 
 #查看交换分区，swap 显示为0 或者交换显示为0 表示关闭了
free -h

关闭防火墙

#禁止
ufw disable 
#查看状态
ufw status 
#inactive 表示禁止了
Status: inactive

安装

按照阿里云安装最新的：

ubuntu-k8s

apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl

你也可以通过指令查看可以安装的版本

apt-cache madison kubelet| awk '{ print $3 }'

centos-k8s

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet

初始化节点

主节点

导入默认初始化配置文件

kubeadm config print init-defaults > kubeadm.yaml

修改配置kubeadm.yaml

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.10.37  #节点局域网ip地址
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: k8s-master
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.28.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12 #默认
  podSubnet: 10.244.0.0/16    #pod子网
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
# KubeProxyConfiguration 代理模式，指定 ipvs，默认是 iptables，iptables 效率低。
kind: KubeProxyConfiguration
mode: ipvs #指定
---
apiVersion: kubelet.config.k8s.io/v1beta1
# 修改 KubeletConfiguration 驱动为 systemd
kind: KubeletConfiguration
cgroupDriver: systemd #指定

执行初始化

kubeadm init --config kubeadm.yaml

此时会输出

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.10.37:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:ecbe8d9e29a5a255ab641d7b95fc643458b5575c33b057b6d56044f395ec92e2 

可以完全按照提示的内容进行集群的操作

执行第一步就可以获取到节点了

root@k8s-master:~# kubectl get nodes
NAME         STATUS     ROLES           AGE     VERSION
k8s-master   NotReady   control-plane   6m56s   v1.28.0

查看系统pod状态，发现coredns处于Pending 状态，原因是缺少pod网络插件

root@k8s-master:~# kubectl get pods -n kube-system -o wide
NAME                             READY   STATUS    RESTARTS   AGE     IP               NODE     NOMINATED NODE   READINESS GATES
coredns-7f8cbcb969-9j5v6         0/1     Pending   0          9m38s   <none>           <none>   <none>           <none>
coredns-7f8cbcb969-vnz5h         0/1     Pending   0          9m38s   <none>           <none>   <none>           <none>
etcd-master                      1/1     Running   0          10m     192.168.17.130   master   <none>           <none>
kube-apiserver-master            1/1     Running   0          10m     192.168.17.130   master   <none>           <none>
kube-controller-manager-master   1/1     Running   0          10m     192.168.17.130   master   <none>           <none>
kube-proxy-5hv9x                 1/1     Running   0          9m39s   192.168.17.130   master   <none>           <none>
kube-proxy-gv5g8                 1/1     Running   0          5m21s   192.168.17.132   node2    <none>           <none>
kube-proxy-smk2m                 1/1     Running   0          6m18s   192.168.17.131   node1    <none>           <none>
kube-scheduler-master            1/1     Running   0          10m     192.168.17.130   master   <none>           <none>

接下来安装pod网络插件

链接：https://pan.baidu.com/s/1eHV4KuM_1bUTuZ_2zW3UQg?pwd=ypdi 
提取码：ypdi

提取calico.yaml,应用安装，单网卡这样就够了，如果多网卡还要修改calico.yaml文件

kubectl apply -f calico.yaml

至此主节点安装完成，将节点加入集群，执行第三步，

kubeadm join 192.168.10.37:6443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:ecbe8d9e29a5a255ab641d7b95fc643458b5575c33b057b6d56044f395ec92e2 

如果忘记，可以从主节点重新获取

kubeadm token create --print-join-command

可能会遇到的问题

1

Failed to create pod sandbox: open /run/systemd/resolve/resolv.conf: no such file or directory

将主节点/run/systemd/resolve 目录下的文件 ，拷贝到节点一样的目录下，没有则自己创建

2

/proc/sys/net/bridge/bridge-nf-call-iptables does not exist

执行如下命令解决

modprobe br_netfilter
echo 1 > /proc/sys/net/ipv4/ip_forward

3

Failed to create pod sandbox: open /run/systemd/resolve/resolv.conf: no such file or directory

1、将主节点/run/systemd/resolve 目录下的文件 ，拷贝到节点一样的目录下，节点没有则自己创建
2、如果没有安装systemd-resolved，则可以安装
3、如果systemd-resolved没有启动则启动他

6443 连接不上

The connection to the server 192.168.10.37:6443 was refused - did you specify the right host or port

查看/etc/containerd/config.toml中，修改SystemdCgroup配置为true。

一些常用的指令

删除节点
先驱逐

kubectl drain --ignore-daemonsets <节点名称>

再删除

kubectl delete node <节点名称>

清除ipvs命令

ipvsadm -C

清楚iptables

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X