RockyLinux9.2安装k8s 1.27+calico+BGP+OpenELB

1. 概述

---

本文在rockylinux 9.2 中使用kubeadm部署 Kubernetes 1.27、containerd、calico、BGP等;

使用OpenELB作为LoadBalancer;

使用BIRD模拟物理路由器;

使用kube-vip实现control-plane高可用;

本文所有k8s相关组件都固定版本安装，避免因版本更新导致各种问题；如kubelet-1.27.2、kubeadm-1.27.2、kubectl-1.27.2、calico-3.25.1、calicoctl-3.24.6、containerd-1.6.21等

2. 环境说明

序号	CPU	内存(G)	操作系统	IP	主机名	备注
1	2	12	Rockylinux 9.2	192.168.3.51	bgp-k8s-01.tiga.cc	master
2	2	12	Rockylinux 9.2	192.168.3.52	bgp-k8s-02.tiga.cc	master
3	2	12	Rockylinux 9.2	192.168.3.53	bgp-k8s-03.tiga.cc	master
4	2	12	Rockylinux 9.2	192.168.3.54	bgp-k8s-04.tiga.cc	worker
5	2	12	Rockylinux 9.2	192.168.3.55	bgp-k8s-05.tiga.cc	worker
6	2	12	Rockylinux 9.2	192.168.3.56	bgp-k8s-06.tiga.cc	worker
7	2	12	Rockylinux 9.2	192.168.3.57	bgp-k8s-07.tiga.cc	worker
8	2	12	Rockylinux 9.2	192.168.3.58	bgp-k8s-08.tiga.cc	worker
9	2	2	Rockylinux 9.2	192.168.3.61	bird-01.tiga.cc	bird(模拟路由器)

3. 准备工作

3.1 检查mac和product_uuid

---

同一个k8s集群内的所有节点需要确保mac地址和product_uuid均唯一，部署前需检查信息

# 检查mac地址
ip ad

# 检查product_uuid
cat /sys/class/dmi/id/product_uuid

3.2 修改host文件

---

echo '192.168.3.50 bgp-k8s-api-server.tiga.cc' >> /etc/hosts
echo '192.168.3.51 bgp-k8s-01.tiga.cc' >> /etc/hosts
echo '192.168.3.52 bgp-k8s-02.tiga.cc' >> /etc/hosts
echo '192.168.3.53 bgp-k8s-03.tiga.cc' >> /etc/hosts
echo '192.168.3.54 bgp-k8s-04.tiga.cc' >> /etc/hosts
echo '192.168.3.55 bgp-k8s-05.tiga.cc' >> /etc/hosts
echo '192.168.3.56 bgp-k8s-06.tiga.cc' >> /etc/hosts
echo '192.168.3.57 bgp-k8s-07.tiga.cc' >> /etc/hosts
echo '192.168.3.58 bgp-k8s-08.tiga.cc' >> /etc/hosts
echo '192.168.3.61 bird-01.tiga.cc'    >> /etc/hosts

3.3 关闭firewalld

---

systemctl disable firewalld
systemctl stop firewalld

3.4 关闭swap

---

sed -i 's:/dev/mapper/rl-swap:#/dev/mapper/rl-swap:g' /etc/fstab

3.5 关闭selinux

---

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0

3.6 安装ipvs

---

yum install -y ipvsadm

3.7 开启路由转发

echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf
sysctl -p

3.8 加载bridge

yum install -y epel-release
yum install -y bridge-utils

modprobe br_netfilter
echo 'br_netfilter' >> /etc/modules-load.d/bridge.conf
echo 'net.bridge.bridge-nf-call-iptables=1' >> /etc/sysctl.conf
echo 'net.bridge.bridge-nf-call-ip6tables=1' >> /etc/sysctl.conf
sysctl -p

4. 安装containerd

---

官方文档: https://github.com/containerd/containerd/blob/main/docs/getting-started.md

4.1 安装containerd

---

yum install -y yum-utils

# DEB 和 RPM 格式的 containerd.io 包由 Docker（而不是 containerd 项目）分发
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# 查看yum源中所有containerd版本
# yum list containerd.io --showduplicates | sort -r

yum install -y containerd.io-1.6.21

systemctl start containerd
systemctl enable containerd

确认containerd是否安装成功

containerd -v

输出

containerd containerd.io 1.6.21 3dce8eb055cbb6872793272b4f20ed16117344f8

设置crictl的CRI endpoint为containerd

echo 'runtime-endpoint: unix:///run/containerd/containerd.sock' >> /etc/crictl.yaml
echo 'image-endpoint: unix:///run/containerd/containerd.sock' >> /etc/crictl.yaml
echo 'timeout: 10' >> /etc/crictl.yaml
echo 'debug: false' >> /etc/crictl.yaml

4.2 安装cni-plugins (可选)

---

如果使用yum安装kubelet会自动安装kubernetes-cni，无需执行本步骤

使用yum源安装containerd的方式会把runc安装好，但是并不会安装cni-plugins，还需要手动安装cni-plugins。

wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz

mkdir -p /opt/cni/bin
tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.3.0.tgz

确认cni-plugins是否安装成功

/opt/cni/bin/host-local

输出

CNI host-local plugin v1.3.0
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0

4.3 配置cgroup driver

4.3.1 确认系统当前的cgroup版本

stat -fc %T /sys/fs/cgroup/

如果是cgroup v2，则输出: cgroup2fs
如果是cgroup v1，则输出: tmps

支持cgroup v2需要Linux Kernel 5.8或者更高；需要containerd v1.4或者更高。

4.3.2 containerd配置cgroup driver

---

修改配置文件

# 配置文件说明: https://github.com/containerd/containerd/blob/main/docs/man/containerd-config.toml.5.md
containerd config default > /etc/containerd/config.toml
# 启用systemd cgroup
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
systemctl restart containerd

5. 安装kubelet、kubectl、kubeadm

---

官方文档: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

5.1 添加yum源

---

# 注意，这里就是用el7的源，google没有为rhel8、rhel9再单独打包
cat <<EOF |  tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=kubernetes
baseurl=https://mirrors.tuna.tsinghua.edu.cn/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
EOF

5.2 安装kubelet、kubectl、kubeadm

---

• kubelet所有节点都需要安装
• kubectl可以安装在任意机器，只要能远程连接到k8s的节点即可
• kubeadm所有节点都需要安装

# 安装yum源中最新版本
# yum install -y kubelet kubeadm kubectl

# 查看当前yum源有哪些kubelet版本
# yum list kubelet kubeadm kubectl  --showduplicates

# yum 安装指定1.27.2版本
yum install -y kubelet-1.27.2-0  kubeadm-1.27.2-0  kubectl-1.27.2-0 

systemctl enable kubelet
systemctl start kubelet

6. 初始化集群

6.1 控制平面高可用(kube-vip)

---

官方文档: https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#kube-vip

kube-vip是一个 keepalived 和 haproxy 的更“传统”方法的替代方案，kube-vip 在一项服务中实现了虚拟 IP 的管理和负载均衡。它可以在第 2 层（使用 ARP 和 leaderElection ）或第 3 层使用 BGP 对等实现。kube-vip 将在控制平面节点上作为静态 pod 运行。

export VIP=192.168.3.50
export INTERFACE='enp1s0'

# KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")