提示:文章写完后,目录可以自动生成,如何生成可参考右边的帮助文档
前言
由于k8s的应用越来越广泛,使用越来越多,我本人也是被迫在学习。今天实验下用新版本的linux–相当于Centos9,安装下新版本的kubenetes 1.31。环境不稳定,仅用来学习模拟,不建议用于生产。
一、系统安装完后初始配置
官方文档可参加如下地址:
https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/
大神的操作文档:
https://gitlab-k8s.xuxiaowei.com.cn/gitlab-k8s/docs/k8s/centos-install.html
三台主机的ip配置
10.9.1.235 master
10.9.1.236 node1
10.9.1.237 node2
1.1 准备工作
sudo yum -y install vim
sudo yum -y install wget
sudo yum -y install bash-completion
# cat /etc/hosts
10.9.1.235 master
10.9.1.236 node1
10.9.1.237 node2
source /etc/profile
sudo systemctl stop firewalld.service
sudo systemctl disable firewalld.service
禁用selinux
SELINUX=disabled
关闭交换分区
free -h
sudo swapoff -a
sudo sed -i 's/.*swap.*/#&/' /etc/fstab
free -h
# 设置所需的 sysctl 参数,参数在重新启动后保持不变
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1 #可能这个参数不生效了
net.bridge.bridge-nf-call-ip6tables = 1 #可能这个参数不生效了
net.ipv4.ip_forward = 1
EOF
# 应用 sysctl 参数而不重新启动
sudo sysctl --system
1.2 安装docker 、Containerd
Docker 不是必须的,k8s 1.24.0 开始使用 Containerd 替代 Docker,此处用docker仅仅是用来下载镜像用。
# step 1: 安装必要的一些系统工具
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
# Step 2: 添加软件源信息
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# Step 3
sudo sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo
# Step 4: 更新并安装Docker-CE
sudo yum makecache fast
# 搜索 docker-ce 版本号
# yum --showduplicates list docker-ce
# 搜索 docker-ce-cli 版本号
# yum --showduplicates list docker-ce-cli
# 安装 docker containerd
sudo yum install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# 启动 docker 时,会启动 containerd
# sudo systemctl status containerd.service --no-pager
sudo systemctl stop containerd.service
修改下配置文件
sudo cp /etc/containerd/config.toml /etc/containerd/config.toml.bak
sudo containerd config default > $HOME/config.toml
sudo cp $HOME/config.toml /etc/containerd/config.toml
# 修改 /etc/containerd/config.toml 文件后,要将 docker、containerd 停止后,再启动
sudo sed -i "s#registry.k8s.io/pause#registry.aliyuncs.com/google_containers/pause#g" /etc/containerd/config.toml
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.10" 这个要改为3.10
# https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/#containerd-systemd
# 确保 /etc/containerd/config.toml 中的 disabled_plugins 内不存在 cri
sudo sed -i "s#SystemdCgroup = false#SystemdCgroup = true#g" /etc/containerd/config.toml
# containerd 忽略证书验证的配置
# [plugins."io.containerd.grpc.v1.cri".registry.configs]
# [plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.0.12:8001".tls]
# insecure_skip_verify = true
sudo systemctl enable --now containerd.service
# sudo systemctl status containerd.service --no-pager
# sudo systemctl status docker.service --no-pager
sudo systemctl start docker.service
# sudo systemctl status docker.service --no-pager
sudo systemctl enable docker.service
sudo systemctl enable docker.socket
sudo systemctl list-unit-files | grep docker
sudo mkdir -p /etc/docker
配置镜像加速网址
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://hnkfbj7x.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo docker info
sudo systemctl status docker.service --no-pager
sudo systemctl status containerd.service --no-pager
# 开启 crictl 配置(可选)
# 安装完 k8s 后,才有 crictl 命令
# 参考:
# GitHub:https://github.com/containerd/containerd/blob/main/docs/cri/crictl.md
# 作者镜像仓库:https://framagit.org/mirrors-github/containerd/containerd/-/blob/main/docs/cri/crictl.md
cat <<EOF > /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true
EOF
1.3 安装k8s
添加yum仓库
# 此操作会覆盖 /etc/yum.repos.d/kubernetes.repo 中现存的所有配置
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF
安装软件
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet
# https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
# 设置所需的 sysctl 参数,参数在重新启动后保持不变
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# 应用 sysctl 参数而不重新启动
sudo sysctl --system
# https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/
# 通过运行以下指令确认 br_netfilter 和 overlay 模块被加载:
lsmod | grep br_netfilter
lsmod | grep overlay
# https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/
# 通过运行以下指令确认 net.bridge.bridge-nf-call-iptables、net.bridge.bridge-nf-call-ip6tables 和 net.ipv4.ip_forward 系统变量在你的 sysctl 配置中被设置为 1:
sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward
此时kubelet未初始化,会一直存在报错
journalctl -xefu kubelet
sudo systemctl status kubelet
然后克隆出别外两台虚拟机,别外两台的机器ID,需要重置
cat /dev/null > /var/lib/dbus/machine-id
rm -rf /etc/machine-id
systemd-machine-id-setup
cat /etc/machine-id > /var/lib/dbus/machine-id
1.4 k8s集群初始化
# 初始化前:列举所有所需的镜像,使用阿里云镜像
kubeadm config images list --image-repository=registry.aliyuncs.com/google_containers
registry.aliyuncs.com/google_containers/kube-apiserver:v1.31.0
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.31.0
registry.aliyuncs.com/google_containers/kube-scheduler:v1.31.0
registry.aliyuncs.com/google_containers/kube-proxy:v1.31.0
registry.aliyuncs.com/google_containers/coredns:v1.11.1
registry.aliyuncs.com/google_containers/pause:3.10
registry.aliyuncs.com/google_containers/etcd:3.5.15-0
# 初始化前:拉取所有的镜像,使用阿里云镜像
kubeadm config images pull --image-repository=registry.aliyuncs.com/google_containers
#kubeadm init --image-repository=registry.aliyuncs.com/google_containers
# 指定集群的IP
kubeadm init --image-repository=registry.aliyuncs.com/google_containers --apiserver-advertise-address=10.9.1.235 --pod-network-cidr=172.18.0.0/16 --service-cidr=10.96.0.0/16 --kubernetes-version v1.31.0
# --apiserver-advertise-address:API 服务器所公布的其正在监听的 IP 地址。如果未设置,则使用默认网络接口。存在多个网卡时推荐设置此参数
# --pod-network-cidr:指明 pod 网络可以使用的 IP 地址段。如果设置了这个参数,控制平面将会为每一个节点自动分配 CIDRs。
# --service-cidr:默认值:"10.96.0.0/12",为服务的虚拟 IP 地址另外指定 IP 地址段
如下显示则表示集群创建成功:
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.9.1.235:6443 --token u70i9j.5kee1b48d8dt45co \
--discovery-token-ca-cert-hash sha256:cf6b53e0ad1d16baa06e2ae6f815a7cb04667bc77c01e6a8d26d721b8e575992
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 或者在环境变量文件 /etc/profile 中添加:export KUBECONFIG=/etc/kubernetes/admin.conf
# 添加环境变量的命令:echo 'export KUBECONFIG=/etc/kubernetes/admin.conf' >>/etc/profile
# 添加完环境变量后,刷新环境变量:source /etc/profile
kubectl cluster-info
# 初始化失败后,可进行重置,重置命令:kubeadm reset
# 执行成功后,会出现类似下列内容:
kubeadm join 10.9.1.235:6443 --token u70i9j.5kee1b48d8dt45co \
--discovery-token-ca-cert-hash sha256:cf6b53e0ad1d16baa06e2ae6f815a7cb04667bc77c01e6a8d26d721b8e575992
# 生成 node 节点加入集群的命令
# kubeadm token create --print-join-command
集群状态和节点状态
kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady control-plane 5m6s v1.31.0
node1 NotReady <none> 34s v1.31.0
node2 NotReady <none> 11s v1.31.0
[root@master ~]# kubectl cluster-info
Kubernetes control plane is running at https://10.9.1.235:6443
CoreDNS is running at https://10.9.1.235:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
1.5 k8s网络插件安装
# 下载
wget --no-check-certificate https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/calico.yaml
# 修改 calico.yaml 文件
vim calico.yaml
# 在 - name: CLUSTER_TYPE 下方添加如下内容
- name: CLUSTER_TYPE
value: "k8s,bgp"
# 下方为新增内容
# 如果集群服务器中存在不同的网卡名称,需要在这里将每台服务器所使用的网卡名称全部填写(使用英文逗号分隔),否则网络无法使用,一直报错
# 例如:集群一共存在10台机器,其中有些机器的网卡名称是 ens33,有些是 eth0,有些是 enp9s0f0,则网卡配置为 interface=ens33,eth0,enp9s0f0
- name: IP_AUTODETECTION_METHOD
value: "interface=ens192"
# 下面是使用命令修改 calico.yaml 文件
# INTERFACE_NAME=ens33
# sed -i '/k8s,bgp/a \ - name: IP_AUTODETECTION_METHOD\n value: "interface=INTERFACE_NAME"' calico.yaml
# sed -i "s#INTERFACE_NAME#$INTERFACE_NAME#g" calico.yaml
修改文件中的 CALICO_IPV4POOL_CIDR,设置为适合您的网络环境的子网掩码(例如 "172.18.0.0/16")。这是 Calico 分配给 Pods 的 IP 地址范围。
上下对齐,注意格式
# 配置网络
kubectl apply -f calico.yaml
控制面板:查看 pods、nodes
如果calico镜像拉取不下来,可手工用镜像拉取命令(**3个node都要拉!3个node都要拉!3个node都要拉!**):
# node类型pod中3个
ctr -n=k8s.io image pull docker.1panel.live/calico/cni:v3.28.1
ctr -n=k8s.io image pull docker.1panel.live/calico/pod2daemon-flexvol:v3.28.1
ctr -n=k8s.io image pull docker.1panel.live/calico/node:v3.28.1
# node-driver类型pod中2个
ctr -n=k8s.io image pull docker.1panel.live/calico/csi:v3.28.1
ctr -n=k8s.io image pull docker.1panel.live/calico/node-driver-registrar:v3.28.1
# typha类型pod中1个
ctr -n=k8s.io image pull docker.1panel.live/calico/typha:v3.28.1
# controller类型pod中1个
ctr -n=k8s.io image pull docker.1panel.live/calico/kube-controllers:v3.28.1
# 这是后来又出现的calico-apiserver 命名空间中需要的
ctr -n=k8s.io image pull docker.1panel.live/calico/apiserver:v3.28.1
kubectl get nodes -o wide
kubectl get pods --all-namespaces -o wide
如果状态都OK,说明部署完成啦
至此,k8s安装与配置已完成,接下来内容就是测试等等的啦。
总结
先大致安装一遍,再慢慢考虑其底层原理,存储,网络等等的。谢谢各们客官,如果这篇文章对你有帮忙,请点个赞,谢谢。
很多都是参考 xuxiaowei 的博文,原文章,请参考 https://gitlab-k8s.xuxiaowei.com.cn