kubernetes集群部署

环境准备

主机名	ip	角色
harbor.timinglee.org	172.25.254.254	harbor仓库
k8s-master.timinglee.org	172.25.254.100	master，k8s集群控制节点
k8s-node1.timinglee.org	172.25.254.10	worker，k8s集群工作节点
k8s-node2.timinglee.org	172.25.254.20	worker，k8s集群工作节点

• 所有节点禁用selinux和防火墙
• 所有节点同步时间和解析
• 所有节点安装docker-ce
• 所有节点禁用swap，注意注释掉/etc/fstab文件中的定义

集群环境初始化
1.禁用所有的swap和开启本地解析

systemctl mask dev-nvme0n1p3.swap
# swapoff -a
# vim /etc/fstab

#
# /etc/fstab
# Created by anaconda on Sun Feb 19 17:38:40 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/rhel-root   /                       xfs     defaults        0 0
UUID=ddb06c77-c9da-4e92-afd7-53cd76e6a94a /boot                   xfs     defaults        0 0
#/dev/mapper/rhel-swap   swap                    swap    defaults        0 0
/dev/cdrom      /media  iso9660 defaults        0 0




[root@k8s-master ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.25.254.100      k8s-master.timinglee.org
172.25.254.10       k8s-node1.timinglee.org
172.25.254.20       k8s-node2.timinglee.org
172.25.254.254      reg.timinglee.org

2.为所有的都安装docker

[root@k8s-master ~]# vim /etc/yum.repos.d/docker.repo
[docker]
name=docker
baseurl=https://mirrors.aliyun.com/docker-ce/linux/rhel/9/x86_64/stable/
gpgcheck=0

[root@k8s-master ~]# dnf install docker-ce -y

3.所有节点设定docker的资源管理模式为systemd

[root@k8s-master ~]# vim /etc/docker/daemon.json
{
        "registry-mirrors": ["https://reg.westos.org"],
}

4.所有阶段复制harbor仓库中的证书并启动docker

[root@k8s-master ~]# ls -l /etc/docker/certs.d/reg.timinglee.org/ca.crt
[root@k8s-master ~]# systemctl enable --now docker

#登陆harbor仓库
[root@k8s-master ~]# docker login reg.timinglee.org
[root@k8s-master ~]# docker info
Client: Docker Engine - Community
 Version:    27.1.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 27.1.2
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd			#资源管理更改为systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.14.0-427.13.1.el9_4.x86_64
 Operating System: Red Hat Enterprise Linux 9.4 (Plow)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 736.3MiB
 Name: k8s-master.timinglee.org
 ID: f3c291bf-287d-4cf6-8e69-5f21c79fa7c6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://reg.westos.org/			#认证harbor仓库
 Live Restore Enabled: false

5.安装K8S部署工具

#部署软件仓库，添加K8S源
[root@k8s-master ~]# vim /etc/yum.repos.d/k8s.repo
[k8s]
name=k8s
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.30/rpm
gpgcheck=0

#安装软件
[root@k8s-master ~]# dnf install kubelet-1.30.0 kubeadm-1.30.0 kubectl-1.30.0 -y

6.设置kubectl命令补齐功能

[root@k8s-master ~]# dnf install bash-completion -y
[root@k8s-master ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc
[root@k8s-master ~]# source  ~/.bashrc

7.在所有节点安装cri-docker
k8s从1.24版本开始移除了dockershim，所以需要安装cri-docker插件才能使用docker安装cri-docker

[root@k8s-master ~]# dnf install libcgroup-0.41-19.el8.x86_64.rpm \
> cri-dockerd-0.3.14-3.el8.x86_64.rpm -y

[root@k8s-master ~]# vim /lib/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket

[Service]
Type=notify

#指定网络插件名称及基础容器镜像
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --pod-infra-container-image=reg.timinglee.org/k8s/pause:3.9

ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl start cri-docker
[root@k8s-master ~]# ll /var/run/cri-dockerd.sock
srw-rw---- 1 root docker 0  8月 26 22:14 /var/run/cri-dockerd.sock		#cri-dockerd的套接字文件

8.在master节点拉取K8S所需镜像

#拉取k8s集群所需要的镜像
[root@k8s-master ~]# kubeadm config images pull \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.30.0 \
--cri-socket=unix:///var/run/cri-dockerd.sock

#上传镜像到harbor仓库
[root@k8s-master ~]# docker images | awk '/google/{ print $1":"$2}' \
| awk -F "/" '{system("docker tag "$0" reg.timinglee.org/k8s/"$3)}'

[root@k8s-master ~]# docker images  | awk '/k8s/{system("docker push "$1":"$2)}'

9.集群初始化

#启动kubelet服务
[root@k8s-master ~]# systemctl status kubelet.service

#执行初始化命令
[root@k8s-master ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 \
--image-repository reg.timinglee.org/k8s \
--kubernetes-version v1.30.0 \
--cri-socket=unix:///var/run/cri-dockerd.sock

#若已经初始化过可用kubeadm reset --cri-socket=unix:///var/run/cri-dockerd.sock进行重置

#指定集群配置文件变量
[root@k8s-master ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

#当前节点没有就绪，因为还没有安装网络插件，容器没有运行
[root@k8s-master ~]# kubectl get node
NAME                       STATUS     ROLES           AGE     VERSION
k8s-master.timinglee.org   NotReady   control-plane   4m25s   v1.30.0
root@k8s-master ~]# kubectl get pod -A
NAMESPACE     NAME                                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-647dc95897-2sgn8                           0/1     Pending   0          6m13s
kube-system   coredns-647dc95897-bvtxb                           0/1     Pending   0          6m13s
kube-system   etcd-k8s-master.timinglee.org                      1/1     Running   0          6m29s
kube-system   kube-apiserver-k8s-master.timinglee.org            1/1     Running   0          6m30s
kube-system   kube-controller-manager-k8s-master.timinglee.org   1/1     Running   0          6m29s
kube-system   kube-proxy-fq85m                                   1/1     Running   0          6m14s
kube-system   kube-scheduler-k8s-master.timinglee.org            1/1     Running   0          6m29s

10.安装flannel网络插件

#下载flannel的yaml部署文件
[root@k8s-master ~]# wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

#现在镜像：
[root@k8s-master ~]# docker pull docker.io/flannel/flannel:v0.25.5
[root@k8s-master ~]# docekr docker.io/flannel/flannel-cni-plugin:v1.5.1-flannel1

#上传镜像到仓库
[root@k8s-master ~]# docker tag flannel/flannel:v0.25.5 \
reg.timinglee.org/flannel/flannel:v0.25.5
[root@k8s-master ~]# docker push reg.timinglee.org/flannel/flannel:v0.25.5

[root@k8s-master ~]# docker tag flannel/flannel-cni-plugin:v1.5.1-flannel1 \
reg.timinglee.org/flannel/flannel-cni-plugin:v1.5.1-flannel1
[root@k8s-master ~]# docker push reg.timinglee.org/flannel/flannel-cni-plugin:v1.5.1-flannel1


#编辑kube-flannel.yml 修改镜像下载位置
[root@k8s-master ~]# vim kube-flannel.yml

#需要修改以下几行
[root@k8s-master ~]# grep -n image kube-flannel.yml
146:        image: reg.timinglee.org/flannel/flannel:v0.25.5
173:        image: reg.timinglee.org/flannel/flannel-cni-plugin:v1.5.1-flannel1
184:        image: reg.timinglee.org/flannel/flannel:v0.25.5

#安装flannel网络插件
[root@k8s-master ~]# kubectl apply -f kube-flannel.yml

11.生成token

[root@k8s-master ~]#   kubeadm token create --print-join-command
kubeadm join 172.25.254.100:6443 --token 5hwptm.zwn7epa6pvatbpwf --discovery-token-ca-cert-hash sha256:52f1a83b70ffc8744db5570288ab51987ef2b563bf906ba4244a300f61e9db23

12.节点扩容
在所有的worker节点中

1 确认部署好以下内容

2 禁用swap

3 安装：

• kubelet-1.30.0
• kubeadm-1.30.0
• kubectl-1.30.0
• docker-ce
• cri-dockerd

4 修改cri-dockerd启动文件添加

• –network-plugin=cni
• –pod-infra-container-image=reg.timinglee.org/k8s/pause:3.9

5 启动服务

• kubelet.service
• cri-docker.service

以上信息确认完毕后即可加入集群

[root@k8s-node1 & 2  ~]# kubeadm join 172.25.254.100:6443 --token 5hwptm.zwn7epa6pvatbpwf --discovery-token-ca-cert-hash sha256:52f1a83b70ffc8744db5570288ab51987ef2b563bf906ba4244a300f61e9db23 --cri-socket=unix:///var/run/cri-dockerd.sock

13.测试集群运行情况

#建立一个pod
[root@k8s-master ~]# kubectl run test --image nginx

#查看pod状态
[root@k8s-master ~]# kubectl get pods
NAME   READY   STATUS    RESTARTS   AGE
test   1/1     Running   0          6m29s

#删除pod
root@k8s-master ~]# kubectl delete pod