Kubernetes集群V1.26二进制部署

Kubernetes集群V1.26二进制部署

1:环境介绍

  • 本环境是基于内网环境无互联网情况下的Kubernetes集群的部署,Runtime使用containerd。亲测有效,有基础可以修改我的配置文件,不懂请不要更改,无脑负责粘贴。

容器编排:Kubernetes允许您在一个集群中运行和管理容器化的应用程序。它可以自动化容器的部署、扩展和运维。
自动化部署:Kubernetes提供了强大的自动化部署功能,可以根据定义的规则自动部署新的应用程序副本。
服务发现与负载均衡:Kubernetes可以自动将网络流量路由到应用程序的正确部分,以及提供负载均衡功能,确保应用程序的高可用性和稳定性。
自我修复:Kubernetes可以自动检测和替换失败的容器实例,确保应用程序持续可用。
水平扩展:Kubernetes允许您根据应用程序的负载自动扩展应用程序副本的数量,以应对流量的变化。
存储编排:Kubernetes提供了灵活的存储解决方案,包括将存储卷挂载到容器内以及动态分配存储。
自定义调度策略:Kubernetes允许您定义自己的调度策略,以满足特定的业务需求。
可扩展性:Kubernetes是一个高度可扩展的平台,可以在各种基础设施上运行,包括公有云、私有云和裸机环境

2:基础架构介绍

选取3台Centos7.9系统作为Master节点配置VIP实现高可用。选取2台Centos7.9系统作为Node节点。共计五台机器作为我们实验的Kubernetes集群

名称 IP地址 系统 组件名称
K8s-master01 10.251.251.71 Centos7.9 Kube-ApiServer、Kube-Controller-Manager、Kube-Scheduler
Etcd、Kubelet、Kube-Proxy、Nfs-Client、haproxy、Keepalived、Nginx
K8s-master02 10.251.251.72 Centos7.9 Kube-ApiServer、Kube-Controller-Manager、Kube-Scheduler
Etcd、Kubelet、Kube-Proxy、Nfs-Client、haproxy、Keepalived、Nginx
K8s-master03 10.251.251.73 Centos7.9 Kube-ApiServer、Kube-Controller-Manager、Kube-Scheduler
Etcd、Kubelet、Kube-Proxy、Nfs-Client、haproxy、Keepalived、Nginx
K8s-node01 10.251.251.74 Centos7.9 Kuberlet、Kube-Proxy、Nfs-Client、Nginx
K8s-node02 10.251.251.75 Centos7.9 Kuberlet、Kube-Proxy、Nfs-Client、Nginx
K8s-master-lb 10.251.251.77 VIP VIP
2.1:资源清单

请添加图片描述

2.2:软件版本
软件 版本
kernel v4.19.12
CentOS 7 v7.9
kube-apiserver、kube-controller-manager、kube-scheduler、kubelet、kube-proxy v1.25.4
etcd v3.5.6
containerd v1.6.10
docker v20.10.21
cfssl v1.6.3
cni v1.1.1
crictl v1.26.15
haproxy v1.8.27
keepalived v2.1.5
calico网络插件 v3.26.4
coredns解析 v1.9.4
merics资源展示 v0.5
dashboard页面 v2.7.0
containerd-runtime资源池 v1.6.8
2.3:下载地址
组件名称 下载地址
Kube-Client下载地址 https://cdn.dl.k8s.io/release/v1.26.15/kubernetes-server-linux-amd64.tar.gz
Docker下载地址 https://download.docker.com/linux/static/stable/x86_64/
Etcd下载地址 https://ghproxy.com/https://github.com/etcd-io/etcd/releases/download/v3.5.6/etcd-v3.5.6-linux-amd64.tar.gz
Centos7内核升级 http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
Cni下载地址 https://objects.githubusercontent.com/github-production-release-asset-2e65be/84575398/34412816-cbca-47a1-a428-9e738f2451d8?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAVCODYLSA53PQK4ZA%2F20240321%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240321T142346Z&X-Amz-Expires=300&X-Amz-Signature=854d653c47a905eb1af871e496054aa7fb2c90bcc1fea914ca39c0cfd64118c3&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=84575398&response-content-disposition=attachment%3B%20filename%3Dcni-plugins-linux-amd64-v1.1.1.tgz&response-content-type=application%2Foctet-stream
calico下载地址 https://raw.githubusercontent.com/projectcalico/calico/v3.26.4/manifests/calico.yaml
coredns下载地址 https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed
merics下载地址 https://github.com/kubernetes-sigs/metrics-server/releases/tag/v0.5.0
dashboard下载地址 https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
containerd下载地址 https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
nginx下载地址 http://nginx.org/download/nginx-1.22.1.tar.gz
2.4:架构图

请添加图片描述

说明!!!!!!!!!!!!说明!!!!!!!!!!!!

  • ALL代表5台机器都要做同样的操作。NO代表只在master01操作
  • ALL代表5台机器都要做同样的操作。NO代表只在master01操作。
  • ALL代表5台机器都要做同样的操作。NO代表只在master01操作。

请添加图片描述

3:前期准备

3.1:关闭防火墙及SElinux(ALL)
[root@k8s-master01 yaml]# systemctl stop firewalld && systemctl disable firewalld		#关闭防火墙,开机禁用防火墙
[root@k8s-master01 yaml]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config	#关闭SElinux
[root@k8s-master01 yaml]# setenforce 0	#查看是否关闭
3.2:关闭swap内存(ALL)
[root@k8s-master01 yaml]# sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab	#添加#号注释swap内容
3.3:设置主机名
hostnamectl set-hostname k8s-master01
hostnamectl set-hostname k8s-master02
hostnamectl set-hostname k8s-master03
hostnamectl set-hostname k8s-node01
hostnamectl set-hostname k8s-node02
3.4:网络配置
1.网卡配置信息:	#内网环境不需要配置DNS
[root@k8s-master01 yaml]# cat /etc/sysconfig/network-scripts/ifcfg-ens192 
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens192
UUID=e7661c85-02f2-46ef-8043-75cd5b3500cd
DEVICE=ens192
ONBOOT=yes
IPADDR=10.251.251.71	#其他4台依次配置IP
NETMASK=255.255.255.0	#掩码
GATEWAY=10.251.251.254	#网关
2.hosts解析:(ALL)		#5台机器配置
[root@k8s-master01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.251.251.71 k8s-master01
10.251.251.72 k8s-master02
10.251.251.73 k8s-master03
10.251.251.77 k8s-master-lb
10.251.251.74 k8s-node01
10.251.251.75 k8s-node02
3.5:工具下载

各位最好将本地的centos7镜像挂载到系统上,做成本地yum源,方便下载一些命令,工具,例如[vim,wget,curl,telnet,netstat]等。如果没有也没事,咱们在自己的笔记本准备一台电脑【前期可以没有,后期必须得有,拉取镜像用】。

1>:有本地源,下载这些工具(ALL)

"vim" "net-tools" "telnet" "gcc" "c++" "curl" "wget" "lrzsz"

2>:没有本地源的话,在自己笔记本的centos7系统配置一些参数使用yum下载

1.编辑yum.conf配置文件
[root@halo ~]# vim /etc/yum.conf
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0	#将0更改为1---0的意思yum下载不保留安装包,1的意思yum下载保留安装包
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=5
bugtracker_url=http://bugs.centos.org/set_project.php?project_id=23&ref=http://bugs.centos.org/bug_report_page.php?category=yum
distroverpkg=centos-release
2.下载命令格式
[root@halo ~]# yum install vim --downloadonly --downloaddir=/root/k8s/package -y
下载完成后会将vim.rpm包存放到/root/k8s/package目录下,将vim.rpm安装包放到k8s集群,使用以下命令安装,这样就不用考虑依赖的问题了。
[root@halo ~]# yum install vim-7.1-1.el7.x86_64.rpm -y
3.6:升级内核(ALL)
1.centos7当前默认内核:
[root@halo ~]# grubby --default-kernel
/boot/vmlinuz-3.10.0-1127.el7.x86_64
2.升级内核到4.19
[root@halo ~]# wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
[root@halo ~]# yum install -y kernel-ml*
[root@halo ~]# grub2-set-default 0		#设置启动内核版本
[root@halo ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
[root@halo ~]# reboot		#重启系统
[root@halo ~]# uname -a		#查看当前系统内核
Linux halo 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux 
3.7:安装ipvsadm(ALL)

根据3.5的逻辑去下载

1.下载ipvsadm
[root@halo ~]# yum install ipvsadm ipset sysstat conntrack libseccomp -y
2.创建启动服务
[root@k8s-master01 ~]# cat >> /etc/modules-load.d/ipvs.conf <<EOF 
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF
[root@k8s-master01 ~]# systemctl restart systemd-modules-load.service
ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 180224  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          176128  1 ip_vs
nf_defrag_ipv6         24576  2 nf_conntrack,ip_vs
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  3 nf_conntrack,xfs,ip_vs
3.8:修改内核参数(ALL)
[root@k8s-master01 ~]# cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384

net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.all.forwarding = 1
EOF
[root@k8s-master01 ~]# sysctl --system

4:互联网机器配置

在你的笔记本电脑必须有一台能连接互联网的机器,做镜像的拉取,docker pull,ctr pull。

4.1:安装docker
#下载地址:下载所需要的docker版本	我的版本是:25.0.0
https://download.docker.com/linux/static/stable/x86_64/docker-25.0.0.tgz
#将下载的tgz包放到/root/k8s/package/目录下,解压,创建系统服务
[root@halo ~]# tar -xvf docker-25.0.0.tgz
[root@halo ~]# rm -rf docker-19.03.9.tgz
[root@halo ~]# cp /root/k8s/package/docker/* /usr/bin/
[root@halo ~]# cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF
[root@halo ~]# systemctl daemon-reload
[root@halo ~]# systemctl restart docker && systemctl enable docker
[root@halo ~]# vim /etc/docker/daemon.json
{
   
  "registry-mirrors": ["https://你的加速地址.mirror.aliyuncs.com"],
  "insecure-registries": ["公司内部harbor地址"]
}
[root@halo ~]# systemctl daemon-reload && systemctl restart docker
4.2:安装containerd
1.下载安装包
[root@halo ~]# wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
[root@halo ~]# wget https://github.com/containerd/containerd/releases/download/v1.6.8/cri-containerd-cni-1.6.8-linux-amd64.tar.gz
2.创建cni所需目录
[root@halo ~]# mkdir -p /etc/cni/net.d /opt/cni/bin 
3.解压cni包
[root@halo ~]# tar xf cni-plugins-linux-amd64-v*.tgz -C /opt/cni/bin/
4.解压
[root@halo ~]# tar -xzf cri-containerd-cni-*-linux-amd64.tar.gz -C /
5.创建系统服务
[root@halo ~]# cat > /etc/systemd/system/containerd.service <<EOF
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF
6.配置containerd所需模块
[root@halo ~]# cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
7.加载配置好的模块
[root@halo ~]# systemctl restart systemd-modules-load.service
8.配置containerd所需的内核
[root@halo ~]# cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
[root@halo ~]# sysctl --system
9.生成配置文件
[root@halo ~]# mkdir -p /etc/containerd
[root@halo ~]# containerd config default | tee /etc/containerd/config.toml

[root@halo ~]# sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
[root@halo ~]# cat /etc/containerd/config.toml | grep SystemdCgroup

#可以更改你的仓库地址
[root@halo ~]# sed -i "s#registry.k8s.io#registry.cn-hangzhou.aliyuncs.com/chenby#g" /etc/containerd/config.toml	
[root@halo ~]# cat /etc/containerd/config.toml | grep sandbox_image		#这是k8s所需的基础镜像

#对照上面的拉取地址,下面操作是跟换镜像源,类似docker的镜像加速器
[root@halo ~]# sed -i "s#config_path\ \=\ \"\"#config_path\ \=\ \"/etc/containerd/certs.d\"#g" /etc/containerd/config.toml	
[root@halo ~]# cat /etc/containerd/config.toml | grep certs.d
[root@halo ~]# mkdir /etc/containerd/certs.d/docker.io -pv
[root@halo ~]# cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://hub-mirror.c.163.com"]
  capabilities = ["pull", "resolve"]
EOF
10.配置circtl客户端(k8s专用命令)
[root@halo ~]# wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.24.2/crictl-v1.24.2-linux-amd64.tar.gz
[root@halo ~]# tar xf crictl-v*-linux-amd64.tar.gz -C /usr/bin/
[root@halo ~]# cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
11.重启containerd,检查配置
[root@halo ~]# systemctl daemon-reload
[root@halo ~]# systemctl restart containerd && systemctl enable --now containerd
[root@halo ~]# crictl info
  },
  "golang": "go1.17.13",
  "lastCNILoadStatus": "OK",
  "lastCNILoadStatus.default": "OK"
}

!!!!!以下操作将是正题,一定要谨慎操作。可以将第3骤做完之后拍个快照。!!!!!

[root@halo ~]# 				互联网机器系统名称

[root@k8s-master01 ~]# 		k8s集群机器系统名称

5:Kubernetes和Etcd安装

5.1:解压安装
######所有的安装包都在/root/package/目录下
[root@k8s-master01 package]# ls
docker  ipvs  kernel  kube  lvs  node_export  system

1.k8s安装
[root@k8s-master01 ~]# tar -xf /root/package/kube/kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
2.etcd安装
[root@k8s-master01 ~]# tar -xf /root/package/kube/kubernetes-server-linux-amd64.tar.gz && mv etcd-*/etcd /usr/local/bin/ && mv etcd-*/etcdctl /usr/local/bin/
3.查看/usr/local/bin/目录
[root@k8s-master01 package]# ls /usr/local/bin/
cfssl      					containerd       			containerd-shim-runc-v1  				containerd-stress  
critest      				ctr   						etcdctl        							kube-controller-manager  
kubelet     				kube-scheduler				cfssljson  								containerd-shim  
containerd-shim-runc-v2 	crictl      	       		ctd-decoder  							etcd  
kube-apiserver  			kubectl  	                kube-proxy
5.1.1:查看版本
[root@k8s-master01 package]# kubelet --version
Kubernetes v1.26.15
[root@k8s-master01 package]# etcdctl version
etcdctl version: 3.5.6
API version: 3.5
5.1.2:分发组件
[root@k8s-master01 package]# Master='k8s-master02 k8s-master03'
[root@k8s-master01 package]# Work='k8s-node01 k8s-node02'
[root@k8s-master01 package]# for NODE in $Master; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done
[root@k8s-master01 package]# for NODE in $Work; do     scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done
[root@k8s-master01 package]# mkdir -p /opt/cni/bin
5.2:证书创建
[root@k8s-master01 package]# mkdir -p /root/pki && cd pki
  • 32
    点赞
  • 21
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值