openEuler 基于 kubeadm 部署k8s

原创已于 2025-08-03 10:36:34 修改 · 1.4k 阅读

24 ·

CC 4.0 BY-SA版权

文章标签：

#kubernetes #docker #容器

于 2025-03-20 16:14:43 首次发布

一、主机配置说明

作用	IP地址	操作系统	配置	关键组件
k8s-master01	192.168.75.144	Rocky Linux release 9	2颗CPU 4G内存 30G硬盘	kube-apiserver, etcd, etc
k8s-node01	192.168.75.146	Rocky Linux release 9	2颗CPU 4G内存 30G硬盘	kubelet, kube-proxy
k8s-node02	192.168.75.147	Rocky Linux release 9	2颗CPU 4G内存 30G硬盘	kubelet, kube-proxy

二、主机准备

1、系统最小化安装

2、替换默认源：将源替换成可用源即可

sed -e 's|^mirrorlist=|#mirrorlist=|g' \
-e 's|^#baseurl=http://dl.rockylinux.org/$contentdir|baseurl=https://mirrors.aliyun.com/rockylinux|g' \
-i.bak \
/etc/yum.repos.d/rocky*.repo

3、安装epel软件仓库，更换国内源

3.1、在 Rocky Linux 9 中启用并安装 EPEL Repo。

dnf config-manager --set-enabled crb
dnf install epel-release

3.2、备份(如有配置其他epel源)并替换为国内镜像

注意最后这个库，阿里云没有对应的镜像，不要修改它，如果误改恢复原版源即可

cp /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel.repo.backup
cp /etc/yum.repos.d/epel-testing.repo /etc/yum.repos.d/epel-testing.repo.backup
cp /etc/yum.repos.d/epel-cisco-openh264.repo /etc/yum.repos.d/epel-cisco-openh264.repo.backup

3.3、将 repo 配置中的地址替换为阿里云镜像站地址

sed -e 's!^metalink=!#metalink=!g' \
-e 's!^#baseurl=!baseurl=!g' \
-e 's!https\?://download\.fedoraproject\.org/pub/epel!https://mirrors.aliyun.com/epel!g' \
-e 's!https\?://download\.example/pub/epel!https://mirrors.aliyun.com/epel!g' \
-i /etc/yum.repos.d/epel{,-testing}.repo

现在我们有了 EPEL 仓库，更新仓库缓存

dnf clean all
dnf makecache

4、配置主机名和ip

分别在对应的主机上配置

hostnamectl set-hostname k8s-master01 #可以按照自己的需要自行改名
hostnamectl set-hostname k8s-node01
hostnamectl set-hostname k8s-node02

5、配置host解析

每台虚拟机上都需要配置

cat >> /etc/hosts << EOF
192.168.75.144 k8s-master01
192.168.75.146 k8s-node01
192.168.75.147 k8s-node02
EOF

# 配置免密登录，只在k8s-master01上操作
ssh-keygen -f ~/.ssh/id_rsa -N '' -q

# 点拷贝秘钥到其他 2 台节点
ssh-copy-id k8s-node01
ssh-copy-id k8s-node02

6、防火墙与SELinux

systemctl disable --now firewalld
sed -i '/^SELINUX=/ c SELINUX=disabled' /etc/selinux/config
setenforce 0

7、时间同步配置

dnf install -y chrony
# 修改同步服务器
sed -i '/^pool/ c pool ntp1.aliyun.com iburst' /etc/chrony.conf
systemctl restart chronyd
systemctl enable chronyd

[root@localhost ~]# chronyc sources
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* 47.96.149.233 2 6 17 3 -772us[-4593us] +/- 35ms

8、配置内核转发及网桥过滤

添加网桥过滤及内核转发配置文件
# cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness = 0
EOF

加载br_netfilter模块
# modprobe br_netfilter

查看是否加载
# lsmod | grep br_netfilter
br_netfilter 22256 0
bridge 151336 1 br_netfilter

使用新添加配置文件生效
# sysctl -p /etc/sysctl.d/k8s.conf

9、关闭swap

永远关闭swap分区
sed -i 's/.*swap.*/#&/' /etc/fstab

10、启用ipvs

cat >> /etc/modules-load.d/ipvs.conf << EOF
br_netfilter
ip_conntrack
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF

dnf install ipvsadm ipset sysstat conntrack libseccomp -y

# 重启服务
systemctl restart systemd-modules-load.service
# 查看以下内容
# lsmod | grep -e ip_vs -e nf_conntrack

11、句柄数最大

ulimit -SHn 65535
cat >> /etc/security/limits.conf <<EOF
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* seft memlock unlimited
* hard memlock unlimitedd
EOF
查看修改结果
ulimit -a

12、系统优化

cat > /etc/sysctl.d/k8s_better.conf << EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF

modprobe br_netfilter
lsmod |grep conntrack
modprobe ip_conntrack
sysctl -p /etc/sysctl.d/k8s_better.conf

三、容器运行时工具安装及运行

1、安装docker

# Step 1: 安装依赖
yum install -y yum-utils device-mapper-persistent-data lvm2

# Step 2: 添加软件源信息
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/rhel/docker-ce.repo

# Step 3: 更新并安装Docker-CE
yum -y install docker-ce

# docker -v
Docker version 27.5.1, build 9f9e405

# 设置国内镜像加速
mkdir -p /etc/docker/
cat >> /etc/docker/daemon.json << EOF
{
"registry-mirrors":["https://p3kgr6db.mirror.aliyuncs.com",
"https://docker.m.daocloud.io",
"https://your_id.mirror.aliyuncs.com",
"https://docker.nju.edu.cn/",
"https://docker.anyhub.us.kg",
"https://dockerhub.jobcher.com",
"https://dockerhub.icu",
"https://docker.ckyl.me",
"https://cr.console.aliyun.com"
],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

设置docker开机启动并启动
# systemctl enable --now docker

查看docker版本
# docker version

2、安装cri-dockerd

1. 下载最新版cri-dockerd rpm包

网络条件好的话直接使用wget下载，网络条件一般的话可以在github上面先下载再上传到虚拟机

下载地址：Releases · Mirantis/cri-dockerd (github.com)。

https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.16/cri-dockerd-0.3.16-3.fc35.x86_64.rpm

2、安装cri-docker

wget -c https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.16/cri-dockerd-0.3.16-3.fc35.x86_64.rpm
wget -c https://rpmfind.net/linux/almalinux/8.10/BaseOS/x86_64/os/Packages/libcgroup-0.41-19.el8.x86_64.rpm
yum install libcgroup-0.41-19.el8.x86_64.rpm ---先安装
yum install cri-dockerd-0.3.16-3.fc35.x86_64.rpm ---后安装

3、启动cri-docker服务

systemctl enable cri-docker

4、cri-docker 设置国内镜像加速

vim /usr/lib/systemd/system/cri-docker.service
------------------
修改第10行内容
ExecStart=/usr/bin/cri-dockerd --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9 --container-runtime-endpoint fd://
-----------------------------------

# 重启Docker组件
$ systemctl daemon-reload && systemctl restart docker cri-docker.socket cri-docker
# 检查Docker组件状态
$ systemctl status docker cir-docker.socket cri-docker

四、K8S软件安装

1、配置kubernetes源
#添加阿里云YUM软件源
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.32/rpm/repodata/repomd.xml.key
EOF

2、查看所有可用的版本
# yum list kubelet --showduplicates | sort -r |grep 1.32

kubelet.x86_64 1.32.3-150500.1.1 kubernetes
kubelet.x86_64 1.32.2-150500.1.1 kubernetes
kubelet.x86_64 1.32.1-150500.1.1 kubernetes
kubelet.x86_64 1.32.0-150500.1.1 kubernetes
kubelet.src 1.32.3-150500.1.1 kubernetes
kubelet.src 1.32.2-150500.1.1 kubernetes
kubelet.src 1.32.1-150500.1.1 kubernetes
kubelet.src 1.32.0-150500.1.1 kubernetes
kubelet.s390x 1.32.3-150500.1.1 kubernetes
kubelet.s390x 1.32.2-150500.1.1 kubernetes
kubelet.s390x 1.32.1-150500.1.1 kubernetes
kubelet.s390x 1.32.0-150500.1.1 kubernetes
kubelet.ppc64le 1.32.3-150500.1.1 kubernetes
kubelet.ppc64le 1.32.2-150500.1.1 kubernetes
kubelet.ppc64le 1.32.1-150500.1.1 kubernetes
kubelet.ppc64le 1.32.0-150500.1.1 kubernetes
kubelet.aarch64 1.32.3-150500.1.1 kubernetes
kubelet.aarch64 1.32.2-150500.1.1 kubernetes
kubelet.aarch64 1.32.1-150500.1.1 kubernetes
kubelet.aarch64 1.32.0-150500.1.1 kubernetes

3、安装kubelet、kubeadm、kubectl、kubernetes-cni
yum install -y kubelet kubeadm kubectl kubernetes-cni

4、配置cgroup
为了实现docker使用的cgroupdriver与kubelet使用的cgroup的一致性，建议修改如下文件内容。
vim /etc/sysconfig/kubelet #3台全部设置下

添加
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"

设置kubelet为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动

五、K8S集群初始化

# 只在master01节点上操作
创建初始化文件 kubeadm-init.yaml

[root@localhost ~]# kubeadm config print init-defaults > kubeadm-init.yaml
修改如下配置：

- advertiseAddress：为控制平面地址，（ Master 主机 IP ）
   advertiseAddress: 1.2.3.4
修改为 advertiseAddress: 192.168.75.144
- criSocket：为 containerd 的 socket 文件地址
   criSocket: unix:///var/run/containerd/containerd.sock
修改为 criSocket: unix:///var/run/cri-dockerd.sock
- name: node 修改node为 k8s-master01
   name: node
修改为 name: k8s-master01

- imageRepository：阿里云镜像代理地址，否则拉取镜像会失败
   imageRepository: registry.k8s.io
修改为：imageRepository: registry.aliyuncs.com/google_containers
- kubernetesVersion：为 k8s 版本
   kubernetesVersion: 1.32.0
修改为：kubernetesVersion: 1.32.2
注意：一定要配置镜像代理，否则会由于防火墙问题导致集群安装失败
文件末尾增加启用ipvs功能
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs

# 根据配置文件启动 kubeadm 初始化 k8s
$ kubeadm init --config=kubeadm-init.yaml --upload-certs --v=6
...

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.90.11:6443 --token abcdef.0123456789abcdef \
   --discovery-token-ca-cert-hash sha256:06faf8d64c03530bf88d1a34eae877b3d446ab5e4f0e071fc96567ccf53b1e70

六、K8S集群工作节点加入

所有的工作节点加入集群
注意：加入集群时需要添加 --cri-socket unix:///var/run/cri-dockerd.sock
kubeadm join 172.16.90.11:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:06faf8d64c03530bf88d1a34eae877b3d446ab5e4f0e071fc96567ccf53b1e70 \
--cri-socket unix:///var/run/cri-dockerd.sock