在欧拉系统上用Kubeadm方式部署（k8s）kubernetes多master高可用集群

道法自然～

已于 2024-09-10 16:21:07 修改

阅读量2.1k

点赞数 34

分类专栏： kubernets 高可用文章标签： kubernetes 容器云原生

于 2024-09-08 21:23:57 首次发布

本文链接：https://blog.csdn.net/richardlygo/article/details/142032585

版权

kubernets 同时被 2 个专栏收录

54 篇文章 4 订阅

订阅专栏

高可用

16 篇文章 0 订阅

订阅专栏

1.1 Kubernetes高可用集群部署
1.1.1 集群架构
1. 高可用拓扑
可以设置 HA 集群：

使用堆叠（stacked）控制平面节点，其中 etcd 节点与控制平面节点共存；

使用外部 etcd 节点，其中 etcd 在与控制平面不同的节点上运行；

在设置 HA 集群之前，应该仔细考虑每种拓扑的优缺点。

2. 堆叠（Stacked） etcd 拓扑

主要特点：

etcd 分布式数据存储集群堆叠在 kubeadm 管理的控制平面节点上，作为控制平面的一个组件运行。

每个控制平面节点运行 kube-apiserver，kube-scheduler 和 kube-controller-manager 实例。

kube-apiserver 使用 LB 暴露给工作节点。

每个控制平面节点创建一个本地 etcd 成员（member），这个 etcd 成员只与该节点的 kube-apiserver 通信。这同样适用于本地 kube-controller-manager 和 kube-scheduler 实例。

简单概况：每个 master 节点上运行一个 apiserver 和 etcd, etcd 只与本节点 apiserver 通信。

这种拓扑将控制平面和 etcd 成员耦合在同一节点上。相对使用外部 etcd 集群，设置起来更简单，而且更易于副本管理。

然而堆叠集群存在耦合失败的风险。如果一个节点发生故障，则 etcd 成员和控制平面实例都将丢失，并且冗余会受到影响。可以通过添加更多控制平面节点来降低此风险。应该为 HA 集群运行至少三个堆叠的控制平面节点（防止脑裂）。

这是 kubeadm 中的默认拓扑。当使用 kubeadm init 和 kubeadm join --control-plane 时，在控制平面节点上会自动创建本地 etcd 成员。

3. 外部 etcd 拓扑

主要特点：

具有外部 etcd 的 HA 集群是一种这样的拓扑，其中 etcd 分布式数据存储集群在独立于控制平面节点的其他节点上运行。

就像堆叠的 etcd 拓扑一样，外部 etcd 拓扑中的每个控制平面节点都运行 kube-apiserver，kube-scheduler 和 kube-controller-manager 实例。

同样 kube-apiserver 使用负载均衡器暴露给工作节点。但是，etcd 成员在不同的主机上运行，每个 etcd 主机与每个控制平面节点的 kube-apiserver 通信。

简单概况： etcd 集群运行在单独的主机上，每个 etcd 都与 apiserver 节点通信。

这种拓扑结构解耦了控制平面和 etcd 成员。因此，它提供了一种 HA 设置，其中失去控制平面实例或者 etcd 成员的影响较小，并且不会像堆叠的 HA 拓扑那样影响集群冗余。

但是，此拓扑需要两倍于堆叠 HA 拓扑的主机数量。具有此拓扑的 HA 集群至少需要三个用于控制平面节点的主机和三个用于 etcd 节点的主机。需要单独设置外部 etcd 集群。

1.1.2 基础环境部署
Kubernetes版本：1.28.2

主机	IP地址	操作系统	配置
k8s-master01	192.168.110.21	openEuler-22.03-LTS-SP4	4颗CPU 8G内存 100G硬盘
k8s-master02	192.168.110.22	openEuler-22.03-LTS-SP4	4颗CPU 8G内存 100G硬盘
k8s-master03	192.168.110.23	openEuler-22.03-LTS-SP4	4颗CPU 8G内存 100G硬盘
k8s-node01	192.168.110.24	openEuler-22.03-LTS-SP4	4颗CPU 8G内存 100G硬盘

主机配置

主机名配置

由于本次使用3台主机完成kubernetes集群部署，其中3台为master节点,名称为k8s-master01,k8s-master02,k8s-master03;其中1台为node节点，名称为：k8s-node01

master01节点
# hostnamectl set-hostname k8s-master01 && bash

master02节点
# hostnamectl set-hostname k8s-master02 && bash

master03节点
# hostnamectl set-hostname k8s-master03 && bash

node01节点
# hostnamectl set-hostname k8s-node01 && bash

主机名与IP地址解析

所有集群主机均需要进行配置。


cat >> /etc/hosts <<EOF
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.110.21 k8s-master01
192.168.110.22 k8s-master02
192.168.110.23 k8s-master03
192.168.110.24 k8s-node01
EOF

关闭SWAP分区

所有主机均需要操作。修改完成后需要重启操作系统，如不重启，可临时关闭，

命令为 swapoff -a

#临时关闭
# swapoff -a
 
 
#永远关闭swap分区，需要重启操作系统
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

防火墙配置

所有主机均需要操作。

关闭现有防火墙firewalld
systemctl disable firewalld
 
systemctl stop firewalld
 
firewall-cmd --state
not running

SELINUX配置

所有主机均需要操作。修改SELinux配置需要重启操作系统。

#临时关闭
setenforce 0
 
#永久生效
sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config

时间同步配置

所有主机均需要操作。最小化安装系统需要安装ntpdate软件。

# crontab -l
0 */1 * * * /usr/sbin/ntpdate time1.aliyun.com
 
 
#设置上海时区，东八区
# timedatectl set-timezone Asia/Shanghai

配置内核转发及网桥过滤

所有主机均需要操作。

#开启内核路由转发
sed -i 's/net.ipv4.ip_forward=0/net.ipv4.ip_forward=1/g' /etc/sysctl.conf

#添加网桥过滤及内核转发配置文件
cat <<EOF >/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness = 0
EOF

配置加载br_netfilter模块
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
 
#加载br_netfilter overlay模块
modprobe br_netfilter
modprobe overlay

#查看是否加载
lsmod | grep br_netfilter
br_netfilter           22256  0

使用默认配置文件生效
sysctl -p

#使用新添加配置文件生效
sysctl -p /etc/sysctl.d/k8s.conf

安装ipset及ipvsadm

所有主机均需要操作。

安装ipset及ipvsadm
yum -y install ipset ipvsadm

#配置ipvsadm模块加载方式.添加需要加载的模块
cat > /etc/sysconfig/modules/ipvs.module <<EOF
modprobe -- ip_vs
modprobe -- ip_vs_sh
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- nf_conntrack
EOF

授权、运行、检查是否加载
chmod 755 /etc/sysconfig/modules/ipvs.module &&  /etc/sysconfig/modules/ipvs.module
 
查看对应的模块是否加载成功
# lsmod | grep -e ip_vs -e nf_conntrack_ipv4

Docker环境准备(优先访问：华为欧拉系统安装docker-CSDN博客)

3.2.1 Docker安装环境准备

准备一块单独的磁盘，建议单独把/var/lib/docker 挂载在一个单独的磁盘上，所有主机均需要操作。

#格式化磁盘
$ mkfs.ext4 /dev/sdb
 
#创建docker工作目录
$ mkdir /var/lib/docker
 
#写入挂载信息到fstab中，永久挂载
$ echo "/dev/sdb /var/lib/docker ext4 defaults 0 0" >>  /etc/fstab
 
#使fstab挂载生效
$ mount -a
 
#查看磁盘挂载
$ df -h /dev/sdb

3.2.2 可选（一）：docker容器

#二进制部署docker，下载docker
wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.9.tgz
tar -xf docker-20.10.9.tgz
cp docker/* /usr/bin
which docker

3.2.3 编写docker.service文件

如果需要指定docker工作目录，需要配置 ExecStart=/usr/bin/dockerd --graph=/home/application/docker

vim /etc/systemd/system/docker.service
 
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
 
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
 
[Install]
WantedBy=multi-user.target

添加可执行权限

chmod +x /etc/systemd/system/docker.service

创建docker组

groupadd docker

配置docker加速，修改cgroup方式

/etc/docker/daemon.json 默认没有此文件，需要单独创建

在/etc/docker/daemon.json添加如下内容
 
tee /etc/docker/daemon.json <<-'EOF'
{                    
  "registry-mirrors" : [
    "http://hub-mirror.c.163.com"],
    "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

# 启动docker
systemctl enable docker && systemctl restart docker

#查看docker 版本
[root@k8s-master ~]# docker version
Client:
 Version:           20.10.9
 API version:       1.41
 Go version:        go1.16.8
 Git commit:        c2ea9bc
 Built:             Mon Oct  4 16:03:22 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
 
Server: Docker Engine - Community
 Engine:
  Version:          20.10.9
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.8
  Git commit:       79ea9d3
  Built:            Mon Oct  4 16:07:30 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.11
  GitCommit:        5b46e404f6b9f661a205e28d59c982d3634148f8
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

3.2.5 【可选：container 容器】

 
1.使用containerd 作为容器，下载 containerd 包
 
# wget https://github.com/containerd/containerd/releases/download/v1.6.6/cri-containerd-cni-1.6.6-linux-amd64.tar.gz
 
这里需要制定解压目录为【/】，包自带结构。
# tar zxvf cri-containerd-cni-1.6.6-linux-amd64.tar.gz -C /
 
2.创建容器目录
# mkdir /etc/containerd
 
3.生成容器配置文件
# containerd config default  >> /etc/containerd/config.toml
 
4.配置systemdcgroup 驱动程序
# vim /etc/containerd/config.toml
 
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true
 
5.修改sandbox (pause) image地址
# vim /etc/containerd/config.toml
 
[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.2"
 
6.更新runc，因为cri-containerd-cni-1.6.6-linux-amd64.tar.gz的runc二进制文件有问题，最后说明。这一步很重要 ✰ ✰ ✰ ✰ ✰ ✰ ✰ ✰ ✰ ✰ ✰ ✰
# wget https://github.com/opencontainers/runc/releases/download/v1.1.3/runc.amd64
# mv runc.amd64 /usr/local/sbin/runc
mv：是否覆盖"/usr/local/sbin/runc"？y
# chmod +x /usr/local/sbin/runc
 
7.启动containerd服务
# systemctl start containerd
# systemctl enable containerd

3.2.6 cri-dockerd

项目地址: https://github.com/Mirantis/cri-dockerd

所有节点都安装 cri-dockerd

# 下载
wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.3/cri-dockerd-0.3.3.amd64.tgz
tar -xf cri-dockerd-0.3.3.amd64.tgz
cp cri-dockerd/cri-dockerd /usr/bin/
chmod +x /usr/bin/cri-dockerd
 
# 配置启动文件
cat <<"EOF" > /usr/lib/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket
 
[Service]
Type=notify
 
ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
 
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
 
StartLimitBurst=3
 
StartLimitInterval=60s
 
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
 
TasksMax=infinity
Delegate=yes
KillMode=process
 
[Install]
WantedBy=multi-user.target
EOF
 
# 生成socket 文件
 
cat <<"EOF" > /usr/lib/systemd/system/cri-docker.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service
 
[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
 
[Install]
WantedBy=sockets.target
 
EOF
 
# 启动CRI-DOCKER
systemctl daemon-reload
systemctl start cri-docker
systemctl enable cri-docker
systemctl is-active cri-docker

3.3 kubernetes 1.28.2 集群部署(-)

3.3.1 集群软件及版本说明

	kubeadm	kubelet	kubectl
版本	1.28.2	1.28.2	1.28.2
安装位置	集群所有主机	集群所有主机	集群所有主机
作用	初始化集群、管理集群等	用于接收api-server指令，对pod生命周期进行管理	集群应用命令行管理工具

3.3.2 kubernetes YUM源准备

3.3.2.1 阿里云YUM源【国内主机】

cat >/etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
 
yum clean all && yum makecache

3.3.2 集群软件安装

所有节点均可安装

# 查看所有的可用版本
yum list  kubeadm  kubelet kubectl --showduplicates | sort -r
 
yum install  kubelet-1.28.2 kubeadm-1.28.2 kubectl-1.28.2
 
#安装后查看版本
kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:20:54Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
 
 
设置kubelet为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动
 systemctl enable kubelet --now
 
#此时kubelet状态是activating的，不是active的
systemctl is-active kubelet

3.3.4 配置kubelet

为了实现docker使用的cgroupdriver与kubelet使用的cgroup的一致性，建议修改如下文件内容。

cat <<EOF > /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
EOF

Master 节点部署高可用

只在三台Master上做

安装Nginx和Keepalived

wget -c https://nginx.org/packages/rhel/7/x86_64/RPMS/nginx-1.24.0-1.el7.ngx.x86_64.rpm
yum install nginx-1.24.0-1.el7.ngx.x86_64.rpm keepalived.x86_64 -y

修改 Nginx 配置文件

3台master操作。

3 台 master 节点修改/etc/nginx/nginx.conf 配置文件，在envents位置后面添加stream部分内容：

[root@k8s-master-01 ~]# vim /etc/nginx/nginx.conf 
events {
    worker_connections  1024;
}

stream {
 log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
 access_log /var/log/nginx/k8s-access.log main;
 upstream k8s-apiserver {
   server 192.168.110.21:6443;
   server 192.168.110.22:6443;
   server 192.168.110.23:6443;
 }
 server {
   listen 16443;
   proxy_pass k8s-apiserver;
 }
}

改 Keepalived 配置文件

只在 k8s-master01 上做

覆盖修改 k8s-master01 节点配置文件/etc/keepalived/keepalived.conf

[root@k8s-master01 ~]# cat > /etc/keepalived/keepalived.conf<<EOF
! Configuration File for keepalived
global_defs {
 router_id master1
 script_user root
 enable_script_security
}

vrrp_script check_nginx {
  script "/etc/keepalived/check_nginx.sh"
  interval 3
  fall 3
  rise 2
}

vrrp_instance Nginx {
 state MASTER
 interface ens33
 virtual_router_id 51
 priority 200
 advert_int 1
 authentication {
   auth_type PASS
   auth_pass XCZKXY
 }
 track_script {
   check_nginx
 }
 virtual_ipaddress {
   192.168.110.20/24
 }
}
EOF

编写健康检测脚本

[root@k8s-master01 ~]# cat > /etc/keepalived/check_nginx.sh<<EOF
#!/bin/sh
# nginx down
pid=`ps -C nginx --no-header | wc -l`
if [ $pid -eq 0 ]
then
    systemctl start nginx
  sleep 5
    if [ `ps -C nginx --no-header | wc -l` -eq 0 ]
    then
        systemctl stop nginx
    else
      exit 0
    fi
fi
EOF

[root@k8s-master01 ~]# chmod +x /etc/keepalived/check_nginx.sh
[root@k8s-master01 ~]# scp /etc/keepalived/{check_nginx.sh,keepalived.conf} k8s-master02:/etc/keepalived/
[root@k8s-master01 ~]# scp /etc/keepalived/{check_nginx.sh,keepalived.conf} k8s-master03:/etc/keepalived/

复制文件和脚本，k8s-master02 和 k8s-master03 配置如上，注意字段 state 修改为 BACKUP，降低 priority，例如
k8s-master02 的 priority 值为 150，k8s-master03 的 priority 值为 100。

修改其他两台Master

[root@k8s-master02 ~]# sed -i 's/MASTER/BACKUP/' /etc/keepalived/keepalived.conf 
[root@k8s-master02 ~]# sed -i 's/200/150/' /etc/keepalived/keepalived.conf

[root@k8s-master03 ~]# sed -i 's/MASTER/BACKUP/' /etc/keepalived/keepalived.conf 
[root@k8s-master03 ~]# sed -i 's/200/100/' /etc/keepalived/keepalived.conf

启动 nginx 和 Keepalived

3台master操作

systemctl enable nginx --now
systemctl enable keepalived --now

高可用切换验证

[root@k8s-master01 ~]# ip a | grep 192.168.110.20/24 #如果没有首先查看/var/log/message中是不是网卡名称有错
    inet 192.168.110.20/24 scope global secondary ens33
    
# 模拟Keepalived宕机

[root@k8s-master01 ~]# systemctl stop keepalived

[root@k8s-master02 ~]# ip a | grep 192.168.110.20/24   # VIP漂移到master-02
    inet 192.168.110.20/24 scope global secondary ens33
[root@k8s-master03 ~]# ip a | grep 192.168.110.20/24   # master-02宕机
    inet 192.168.110.20/24 scope global secondary ens33
    
[root@k8s-master03 ~]# ip a | grep 192.168.110.20/24
    inet 192.168.110.20/24 scope global secondary ens33  # VIP漂移到master-03
    
[root@k8s-master01 ~]# systemctl start keepalived.service  # 恢复后正常
[root@k8s-master01 ~]# ip a | grep 192.168.110.20/24
    inet 192.168.110.20/24 scope global secondary ens33
 
#所有节点    
ping -c 2 192.168.110.20   #确保集群内部可以通讯
PING 192.168.110.20 (192.168.110.20) 56(84) bytes of data.
64 bytes from 192.168.110.20: icmp_seq=1 ttl=64 time=1.03 ms
64 bytes from 192.168.110.20: icmp_seq=2 ttl=64 time=2.22 ms

--- 192.168.110.20 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1018ms
rtt min/avg/max/mdev = 1.034/1.627/2.220/0.593 ms

1.1.5 集群初始化

只在master01节点上操作

# 创建初始化文件 kubeadm-init.yaml
[root@k8s-master01 ~]# kubeadm config print init-defaults > kubeadm-init.yaml  
[root@k8s-master01 ~]# sed -i 's/1.2.3.4/192.168.110.21/' kubeadm-init.yaml    # 控制平面地址修改为Master主机
[root@k8s-master01 ~]# sed -i 's/: node/: k8s-master-01/' kubeadm-init.yaml    # 修改名称
[root@k8s-master01 ~]# sed -i '24 a controlPlaneEndpoint: 192.168.110.20:16443' kubeadm-init.yaml   # 添加虚拟IP
[root@k8s-master01 ~]# sed -i 's#registry.k8s.io#registry.aliyuncs.com/google_containers#' kubeadm-init.yaml  # 替换镜像源
[root@k8s-master01 ~]# sed -i 's/1.28.0/1.28.2/' kubeadm-init.yaml   # 替换为安装的斑斑
[root@k8s-master01 ~]# cat >> kubeadm-init.yaml << EOF   # 开启IPVS
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
EOF

注意：如果使用Docker做容器运行时，需要修改套接字文件为unix:///var/run/cri-dockerd.sock

根据配置文件启动 kubeadm 初始化 k8s

[root@k8s-master01 ~]# kubeadm init --config=kubeadm-init.yaml --upload-certs --v=6 --ignore-preflight-errors="FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"

生成如下内容：

# 这里忽略防火墙报的错

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.110.20:16443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:28d3d73a81af5468649d287d7269a3c3eca9aa3990c3897f4db04aac805de38a \
        --control-plane --certificate-key 6f7027ad52defa430f33c5d25e7a0d4c1a96f3c316bb58d564e04f04330e6fb5

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.110.20:16443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:28d3d73a81af5468649d287d7269a3c3eca9aa3990c3897f4db04aac805de38a

在 k8s-master01 节点执行以下命令，为 kubectl 复制初始化生成的 config 文件，在该配置文件中，记录了 API Server 的访问地址，所以后面直接执行 kubectl 命令就可以正常连接到 API Server 中。

[root@k8s-master01 ~]# mkdir -p $HOME/.kube
[root@k8s-master01 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master01 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

1.1.6 其他Master节点加入

在 k8s-master02 和 k8s-master03 创建用于存放证书的文件夹

[root@k8s-master02 ~]# mkdir -p /etc/kubernetes/pki/etcd
[root@k8s-master03 ~]# mkdir -p /etc/kubernetes/pki/etcd

传递证书到 k8s-master02 和 k8s-master03 节点

[root@k8s-master01 ~]# scp /etc/kubernetes/pki/ca.* root@k8s-master02:/etc/kubernetes/pki/
[root@k8s-master01 ~]# scp /etc/kubernetes/pki/ca.* root@k8s-master03:/etc/kubernetes/pki/
   
[root@k8s-master01 ~]# scp /etc/kubernetes/pki/sa.* root@k8s-master02:/etc/kubernetes/pki/  
[root@k8s-master01 ~]# scp /etc/kubernetes/pki/sa.* root@k8s-master03:/etc/kubernetes/pki/ 

[root@k8s-master01 ~]# scp /etc/kubernetes/pki/front-proxy-ca.* root@k8s-master02:/etc/kubernetes/pki/
[root@k8s-master01 ~]# scp /etc/kubernetes/pki/front-proxy-ca.* root@k8s-master03:/etc/kubernetes/pki/
  
[root@k8s-master01 ~]# scp /etc/kubernetes/pki/etcd/ca.* root@k8s-master02:/etc/kubernetes/pki/etcd/
[root@k8s-master01 ~]# scp /etc/kubernetes/pki/etcd/ca.* root@k8s-master03:/etc/kubernetes/pki/etcd/

# 这个文件master和node节点都需要

[root@k8s-master01 ~]# scp  /etc/kubernetes/admin.conf k8s-master02:/etc/kubernetes/ 
[root@k8s-master01 ~]# scp  /etc/kubernetes/admin.conf k8s-master03:/etc/kubernetes/ 
[root@k8s-master01 ~]# scp  /etc/kubernetes/admin.conf k8s-node01:/etc/kubernetes/

其他 master 节点加入集群

[root@k8s-master02 ~]# kubeadm join 192.168.110.20:16443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:28d3d73a81af5468649d287d7269a3c3eca9aa3990c3897f4db04aac805de38a \
--control-plane --certificate-key 6f7027ad52defa430f33c5d25e7a0d4c1a96f3c316bb58d564e04f04330e6fb5 \
--ignore-preflight-errors="FileContent--proc-sys-net-bridge-bridge-nf-call-iptables,FileContent--proc-sys-net-ipv4-ip_forward"\
--cri-socket unix:///var/run/cri-dockerd.sock\
--v=6

To start administering your cluster from this node, you need to run the following as a regular user:

        mkdir -p $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.


[root@k8s-master03 ~]# kubeadm join 192.168.110.20:16443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:28d3d73a81af5468649d287d7269a3c3eca9aa3990c3897f4db04aac805de38a \
--control-plane --certificate-key 6f7027ad52defa430f33c5d25e7a0d4c1a96f3c316bb58d564e04f04330e6fb5 \
--ignore-preflight-errors="FileContent--proc-sys-net-bridge-bridge-nf-call-iptables,FileContent--proc-sys-net-ipv4-ip_forward"\
--cri-socket unix:///var/run/cri-dockerd.sock\
--v=6

To start administering your cluster from this node, you need to run the following as a regular user:

        mkdir -p $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

在 k8s-master02 和 k8s-master03 节点执行以下命令，复制 kubeconfig 文件

[root@k8s-master02 ~]# mkdir -p $HOME/.kube
[root@k8s-master02 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master02 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

[root@k8s-master03 ~]# mkdir -p $HOME/.kube
[root@k8s-master03 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master03 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

1.1.7 工作节点加入集群

注意：Docker做容器运行时需要添加 --cri-socket unix:///var/run/cri-dockerd.sock

[root@k8s-node01 ~]# kubeadm join 192.168.110.20:16443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:28d3d73a81af5468649d287d7269a3c3eca9aa3990c3897f4db04aac805de38a \
--ignore-preflight-errors="FileContent--proc-sys-net-bridge-bridge-nf-call-iptables,FileContent--proc-sys-net-ipv4-ip_forward"\
--cri-socket unix:///var/run/cri-dockerd.sock\
--v=6

# 查看node

[root@k8s-master01 ~]# kubectl get nodes
NAME            STATUS     ROLES           AGE     VERSION
k8s-master-01   NotReady   control-plane   7m19s   v1.28.2
k8s-master-02   NotReady   control-plane   80s     v1.28.2
k8s-master-03   NotReady   control-plane   2m1s    v1.28.2
k8s-node-01     NotReady   <none>          13s     v1.28.2

安装集群网络插件

只在master01上操作

calico安装

wget https://docs.projectcalico.org/v3.16/manifests/calico.yaml
 
 
vim  calico.yaml
................
            - name: CALICO_IPV4POOL_CIDR
              value: "192.168.0.0/16"
................

修改calico-kube-controllers调度到k8s-master01

[root@k8s-master01 ~]# kubectl apply -f calico.yaml

监视kube-system命名空间中pod运行情况
[root@k8s-master01 ~]# watch kubectl get pods -n kube-system

[root@k8s-master01 ~]# kubectl get pod -n kube-system 
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-658d97c59c-8mw6g   1/1     Running   0          5m47s
calico-node-5hmzv                          1/1     Running   0          5m47s
calico-node-5qpwt                          1/1     Running   0          5m47s
calico-node-9r72w                          1/1     Running   0          5m47s
calico-node-xgds5                          1/1     Running   0          5m47s
calico-node-z56q4                          1/1     Running   0          5m47s
calico-node-zcrrw                          1/1     Running   0          5m47s
coredns-66f779496c-9jldk                   1/1     Running   0          13m
coredns-66f779496c-dxsgv                   1/1     Running   0          13m
etcd-k8s-master01                          1/1     Running   0          13m
etcd-k8s-master02                          1/1     Running   0          9m33s
etcd-k8s-master03                          1/1     Running   0          9m39s
kube-apiserver-k8s-master01                1/1     Running   0          13m
kube-apiserver-k8s-master02                1/1     Running   0          9m48s
kube-apiserver-k8s-master03                1/1     Running   0          9m39s
kube-controller-manager-k8s-master01       1/1     Running   0          13m
kube-controller-manager-k8s-master02       1/1     Running   0          9m49s
kube-controller-manager-k8s-master03       1/1     Running   0          9m36s
kube-proxy-565m4                           1/1     Running   0          8m56s
kube-proxy-hfk4w                           1/1     Running   0          8m56s
kube-proxy-n94sx                           1/1     Running   0          8m56s
kube-proxy-s75md                           1/1     Running   0          9m49s
kube-proxy-t5xl2                           1/1     Running   0          9m48s
kube-proxy-vqkxk                           1/1     Running   0          13m
kube-scheduler-k8s-master01                1/1     Running   0          13m
kube-scheduler-k8s-master02                1/1     Running   0          9m48s
kube-scheduler-k8s-master03                1/1     Running   0          9m39s

1.1.9 应用部署验证及访问验证

下载etcdctl客户端工具

[root@k8s-master01 ~]# wget -c https://github.com/etcd-io/etcd/releases/download/v3.4.29/etcd-v3.4.29-linux-amd64.tar.gz
[root@k8s-master01 ~]# tar xf etcd-v3.4.29-linux-amd64.tar.gz -C /usr/local/src/
[root@k8s-master01 ~]# mv /usr/local/src/etcd-v3.4.29-linux-amd64/etcdctl /usr/local/bin/

查看etcd集群健康状态

[root@k8s-master01 ~]# ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --write-out=table --endpoints=k8s-master01:2379,k8s-master02:2379,k8s-master03:2379 endpoint health
+--------------------+--------+-------------+-------+
|      ENDPOINT      | HEALTH |    TOOK     | ERROR |
+--------------------+--------+-------------+-------+
| k8s-master01:2379 |   true | 22.678298ms |       |
| k8s-master02:2379 |   true | 22.823849ms |       |
| k8s-master03:2379 |   true | 28.292332ms |       |
+--------------------+--------+-------------+-------+

查看etcd集群可用列表

[root@k8s-master01 ~]# ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --write-out=table --endpoints=k8s-master01:2379,k8s-master02:2379,k8s-master03:2379 member list
+------------------+---------+---------------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  |     NAME      |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+---------------+-----------------------------+-----------------------------+------------+
| 409450d991a8d0ba | started | k8s-master03 | https://192.168.110.23:2380 | https://192.168.110.23:2379 |      false |
| a1a70c91a1d895bf | started | k8s-master01 | https://192.168.110.21:2380 | https://192.168.110.21:2379 |      false |
| cc2c3b0e11f3279a | started | k8s-master02 | https://192.168.110.22:2380 | https://192.168.110.22:2379 |      false |
+------------------+---------+---------------+-----------------------------+-----------------------------+------------+

查看etcd集群leader状态

[root@k8s-master01 ~]# ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --write-out=table --endpoints=k8s-master01:2379,k8s-master02:2379,k8s-master03:2379 endpoint status
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| k8s-master01:2379 | a1a70c91a1d895bf |   3.5.9 |  5.1 MB |      true |      false |         3 |       5124 |               5124 |        |
| k8s-master02:2379 | cc2c3b0e11f3279a |   3.5.9 |  5.1 MB |     false |      false |         3 |       5124 |               5124 |        |
| k8s-master03:2379 | 409450d991a8d0ba |   3.5.9 |  5.1 MB |     false |      false |         3 |       5124 |               5124 |        |
+--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

3.3.9 验证集群可用性

查看所有的节点
 
[root@k8s-master01 ~]# kubectl get nodes
NAME            STATUS     ROLES           AGE     VERSION
k8s-master-01   NotReady   control-plane   7m19s   v1.28.2
k8s-master-02   NotReady   control-plane   80s     v1.28.2
k8s-master-03   NotReady   control-plane   2m1s    v1.28.2
k8s-node-01     NotReady   <none>          13s     v1.28.2

查看集群健康情况
[root@k8s-master01 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true","reason":""}

1. 部署一个应用
apiVersion: apps/v1    #与k8s集群版本有关，使用 kubectl api-versions 即可查看当前集群支持的版本
kind: Deployment    #该配置的类型，我们使用的是 Deployment
metadata:            #译名为元数据，即 Deployment 的一些基本属性和信息
  name: nginx-deployment    #Deployment 的名称
  labels:        #标签，可以灵活定位一个或多个资源，其中key和value均可自定义，可以定义多组，目前不需要理解
    app: nginx    #为该Deployment设置key为app，value为nginx的标签
spec:            #这是关于该Deployment的描述，可以理解为你期待该Deployment在k8s中如何使用
  replicas: 1    #使用该Deployment创建一个应用程序实例
  selector:        #标签选择器，与上面的标签共同作用，目前不需要理解
    matchLabels: #选择包含标签app:nginx的资源
      app: nginx
  template:        #这是选择或创建的Pod的模板
    metadata:    #Pod的元数据
      labels:    #Pod的标签，上面的selector即选择包含标签app:nginx的Pod
        app: nginx
    spec:        #期望Pod实现的功能（即在pod中部署）
      containers:    #生成container，与docker中的container是同一种
      - name: nginx    #container的名称
        image: nginx:1.7.9    #使用镜像nginx:1.7.9创建container，该container默认80端口可访问
       
kubectl apply -f nginx.yaml
 
2. 访问应用
apiVersion: v1
kind: Service
metadata:
  name: nginx-service    #Service 的名称
  labels:         #Service 自己的标签
    app: nginx    #为该 Service 设置 key 为 app，value 为 nginx 的标签
spec:        #这是关于该 Service 的定义，描述了 Service 如何选择 Pod，如何被访问
  selector:        #标签选择器
    app: nginx    #选择包含标签 app:nginx 的 Pod
  ports:
  - name: nginx-port    #端口的名字
    protocol: TCP        #协议类型 TCP/UDP
    port: 80            #集群内的其他容器组可通过 80 端口访问 Service
    nodePort: 32600   #通过任意节点的 32600 端口访问 Service
    targetPort: 80    #将请求转发到匹配 Pod 的 80 端口
  type: NodePort    #Serive的类型，ClusterIP/NodePort/LoaderBalancer
 
kubectl apply -f service.yaml
 
3. 测试
[root@k8s-master01 ~]# kubectl get pods,svc
NAME                                   READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-7ddbb5f97-h9dhs   1/1     Running   0          49m
 
NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
service/kubernetes      ClusterIP   10.96.0.1      <none>        443/TCP        90m
service/nginx-service   NodePort    10.111.132.6   <none>        80:32600/TCP   48m
 
[root@k8s-master01 ~]# curl -I 10.111.132.6
HTTP/1.1 200 OK
Server: nginx/1.7.9
Date: Wed, 16 Nov 2022 07:47:36 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 23 Dec 2014 16:25:09 GMT
Connection: keep-alive
ETag: "54999765-264"
Accept-Ranges: bytes

3.3.10 k8s其他设置

kubectl 命令自动补齐

yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
kubectl completion bash >/etc/bash_completion.d/kubectl

四、参考
https://www.bilibili.com/video/BV1uY411c7qU?p=3&spm_id_from=333.880.my_history.page.click

https://www.jianshu.com/p/a613f64ccab6

https://i4t.com/5435.html

https://www.bilibili.com/video/BV1gS4y1B7Ut?spm_id_from=333.880.my_history.page.click