Kubeadm部署k8s集群v1.21.2(高可用Master节点)
- Kubeadm部署k8s集群v1.21.2(高可用Master节点)
- 一、集群部署前规划
- 二、修改主机名和修改时区
- 三、进行基本配置(所有节点)
- 四、配置时间同步
- 五、修改iptables相关参数(所有节点)
- 六、加载ipvs相关模块(所有节点)
- 七、管理工具ipvsadm安装(所有节点)
- 八、安装与启动Docker(所有节点)
- 九、安装kubeadm、kubelet、kubectl(所有节点)
- 十、安装haproxy和keepalived(三个master节点)
- 十、部署master1节点(k8s-master-1, 192.168.56.101),要求最少要2核心
- 十一、部署k8s-master-2和k8s-master-3
- 十一、执行集群检查
- 十二、部署网络插件(k8s-master节点, 192.168.56.101)
- 十三、部署k8s-node-1和k8s-node-2
Kubeadm部署k8s集群v1.21.2(高可用Master节点)
之前写了一篇Kubeadm部署k8s集群v1.21.2(单Master节点),
现补上高可用的部署方式
,记录采用kubeadm方式部署高可用master节点的K8S集群v1.21.2。
官方文档:https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
一、集群部署前规划
主机 | 操作系统 | IP |
---|---|---|
k8s-master-1 | Centos7.6 | 192.168.56.101 |
k8s-master-2 | Centos7.6 | 192.168.56.102 |
k8s-master-3 | Centos7.6 | 192.168.56.103 |
k8s-node-1 | Centos7.6 | 192.168.56.104 |
k8s-node-2 | Centos7.6 | 192.168.56.105 |
二、修改主机名和修改时区
hostnamectl set-hostname k8s-master-1 ------192.168.56.101
hostnamectl set-hostname k8s-master-2 ------192.168.56.102
hostnamectl set-hostname k8s-master-3 ------192.168.56.103
hostnamectl set-hostname k8s-node-1 ------192.168.56.104
hostnamectl set-hostname k8s-node-2 ------192.168.56.105
#修改时区(所有结点)
timedatectl set-timezone Asia/Shanghai
三、进行基本配置(所有节点)
#修改/etc/hosts文件
cat >> /etc/hosts << EOF
192.168.56.101 k8s-master-1
192.168.56.102 k8s-master-2
192.168.56.103 k8s-master-3
192.168.56.104 k8s-node-1
192.168.56.105 k8s-node-2
EOF
#关闭swap
swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab
#关闭selinux
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config && setenforce 0
#防火墙配置(网上的文章都是直接关防火墙的
#这里我不关闭防火墙,只是开放端口,因为生产环境一般也是开的)
#K8s之防火墙firewalld
firewall-cmd --permanent --add-masquerade
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.96.0.0/16" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.244.0.0/16" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.56.101" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.56.102" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.56.103" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.56.104" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="192.168.56.105" accept"
firewall-cmd --reload
firewall-cmd --list-all
四、配置时间同步
- ####chrony配置(时间同步主服务器
192.168.56.101
)
如果没有互联网环境,需要修改yum.repos.d中的源,然后通过私库进行下载,nexus3私库环境搭建(maven,yum,apt,nodejs),有互联网直接执行yum即可
如果没有互联网环境,修改yum.repos.d中的源,然后通过私库进行下载`,比如:搭建好私服,然后执行以下操作进行修改yum.repos.d中的源
cd /etc/yum.repos.d/
mkdir bak
mv *.repo bak
##################
cat > /etc/yum.repos.d/centos.repo << EOF
[centos_os]
name=centos_os
baseurl=http://192.168.56.1:8081/nexus3/repository/centos/7/os/x86_64/
enabled=1
gpgcheck=0
[centos_extras]
name=centos_extras
baseurl=http://192.168.56.1:8081/nexus3/repository/centos/7/extras/x86_64/
enabled=1
gpgcheck=0
[centos_update]
name=centos_update
baseurl=http://192.168.56.1:8081/nexus3/repository/centos/7/updates/x86_64/
enabled=1
gpgcheck=0
[centos_centosplus]
name=centos_centosplus
baseurl=http://192.168.56.1:8081/nexus3/repository/centos/7/centosplus/x86_64/
enabled=1
gpgcheck=0
EOF
##################
cat > /etc/yum.repos.d/epel.repo << EOF
[epel]
name=epel
baseurl=http://192.168.56.1:8081/nexus3/repository/epel/pub/epel/7/x86_64/
enabled=1
gpgcheck=0
EOF
##################
cat > /etc/yum.repos.d/nginx.repo << EOF
[epel]
name=epel
baseurl=http://192.168.56.1:8081/nexus3/repository/nginx/packages/centos/7/x86_64/
enabled=1
gpgcheck=0
EOF
##################
cat > /etc/yum.repos.d/remi-safe.repo << EOF
[remi-safe]
name=Safe Remi's RPM repository for Enterprise Linux 7 - x86_64
baseurl=http://192.168.56.1:8081/nexus3/repository/enterprise/7/safe/x86_64/
enabled=1
gpgcheck=0
EOF
##################
cat > /etc/yum.repos.d/docker-ce.repo << EOF
[docker-ce-stable]
name=Docker CE Stable -
baseurl=http://192.168.56.1:8081/nexus3/repository/docker-ce/7/x86_64/stable
enabled=1
gpgcheck=0
EOF
##################
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernets]
name=kubernets
baseurl=http://192.168.56.1:8081/nexus3/repository/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
EOF
##################Mysql5.7
cat > /etc/yum.repos.d/mysql.repo << EOF
[mysql-connectors-community]
name=mysql-connectors-community
baseurl=http://192.168.56.1:8081/nexus3/repository/mysql/yum/mysql-connectors-community/el/7/x86_64/
enabled=1
gpgcheck=0
[mysql-tools-community]
name=mysql-tools-community
baseurl=http://192.168.56.1:8081/nexus3/repository/mysql/yum/mysql-tools-community/el/7/x86_64/
enabled=1
gpgcheck=0
[mysql57-community]
name=mysql57-community
baseurl=http://192.168.56.1:8081/nexus3/repository/mysql/yum/mysql-5.7-community/el/7/x86_64/
enabled=1
gpgcheck=0
EOF
##################Google-chrome
cat > /etc/yum.repos.d/google-chrome.repo << EOF
[google-chrome]
name=google-chrome
baseurl=http://192.168.56.1:8081/nexus3/repository/google-chrome/linux/chrome/rpm/stable/x86_64
enabled=1
gpgcheck=0
EOF
#安装chrony
yum install -y chrony
#注释默认ntp服务器
sed -i 's/^server/#&/' /etc/chrony.conf
#指定上游公共 ntp 服务器,并允许其他节点同步时间
####如果有外网,设置从互联网服务器拉取时间
cat >> /etc/chrony.conf << EOF
server time6.aliyun.com iburst
server 0.asia.pool.ntp.org iburst
server 1.asia.pool.ntp.org iburst
server 2.asia.pool.ntp.org iburst
server 3.asia.pool.ntp.org iburst
allow all
EOF
####如果没有外网,就以以本机时间为基点:
sed -i 's/^#local stratum 10/local stratum 10/' /etc/chrony.conf
cat >> /etc/chrony.conf << EOF
server 192.168.56.101 iburst
allow all
EOF
#重启chronyd服务并设为开机启动:
systemctl enable chronyd && systemctl restart chronyd
#开启网络时间同步功能
timedatectl set-ntp true
- ####chrony配置(其他节点)
其他节点以时间同步主服务器192.168.56.101
为基础,设置从时间同步服务器同步时间
######配置所有其他节点(注意修改IP地址):
#安装chrony:
yum install -y chrony
#注释默认服务器
sed -i 's/^server/#&/' /etc/chrony.conf
#指定内网 master节点为上游NTP服务器
echo server 192.168.56.101 iburst >> /etc/chrony.conf
#重启服务并设为开机启动:
systemctl enable chronyd && systemctl restart chronyd
#所有节点执行chronyc sources命令,查看存在以^*开头的行,说明已经与服务器时间同步
chronyc sources
五、修改iptables相关参数(所有节点)
#RHEL / CentOS 7上的一些用户报告了由于iptables被绕过而导致流量路由不正确的问题。创建/etc/sysctl.d/k8s.conf文件,添加如下内容:
cat <<EOF > /etc/sysctl.d/k8s.conf
vm.swappiness = 0
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
# 使配置生效
modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf
六、加载ipvs相关模块(所有节点)
由于ipvs已经加入到了内核的主干,所以为kube-proxy开启ipvs的前提需要加载以下的内核模块。在所有的Kubernetes节点执行以下脚本:
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
#继续执行脚本
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
七、管理工具ipvsadm安装(所有节点)
yum install ipset ipvsadm -y
八、安装与启动Docker(所有节点)
- ####配置docker yum源
如果是有互联网,直接配置如下的docker yum源
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
如果没有互联网环境,修改yum.repos.d中的源,然后通过私库进行下载(步骤见安装chrony组件的无互联网部分内容)
- ####安装指定版本,这里安装3:20.10.7-3.el7
yum list docker-ce --showduplicates | sort -r
yum install -y docker-ce-3:20.10.7-3.el7.x86_64
- ####配置docker 镜像加速及修改cgroupdriver
有互联网时,采用如下:
mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
],
"registry-mirrors": ["https://uyah70su.mirror.aliyuncs.com"]
}
EOF
如果没有互联网,一般会建自己的私库,把自己的私库地址也带上,只是比上面多了一行,妇如:"insecure-registries": ["192.168.56.1:5000"]
mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
],
"insecure-registries": ["192.168.56.1:5000"],
"registry-mirrors": ["https://uyah70su.mirror.aliyuncs.com"]
}
EOF
- ####Docker默认镜像和容器的存储位置存储在/var/lib/docker,如果你的机器的这个目录磁盘太小,很容易导致磁盘爆满,因此可以在其他大的磁盘路径外链的方式来处理,如下(如不修改磁盘存储位置,此步骤不做),这里我将目录修改成/home/data/docker, 采用ln-s的方式。
## 采用ln -s方式挂载/home目录
mkdir /home/data/docker -p
ln -s /home/data/docker /var/lib/docker
- ####启动docker
systemctl start docker && systemctl enable docker
##出现下面结果说明docker启动成功
[root@k8s-master-1 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
九、安装kubeadm、kubelet、kubectl(所有节点)
- ####配置kubernetes.repo的源,由于官方源国内无法访问,这里使用阿里云yum源
如果是有互联网,直接配置如下
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
如果没有互联网环境,修改yum.repos.d中的源,然后通过通过私库进行下载
- ####在所有节点上安装指定版本 kubelet、kubeadm 和 kubectl
yum list kubeadm --showduplicates | sort -r
yum install -y kubelet-1.21.2-0 kubeadm-1.21.2-0 kubectl-1.21.2-0
- ####同样,kubelet相关的数据默认存储在/var/lib/kubelet,如果需要修改数据存储目录进行ln -s处理,如不修改磁盘存储位置,此步骤不做,我这里修改成了使用/home/data/kubelet目录来存储。
#修改kubelet数据目录
mkdir /home/data/kubelet -p
ln -s /home/data/kubelet /var/lib/kubelet
- ####将kubelet服务设置为自启动,但systemctl start kubelet不需要执行,k8s部署过程中会自动呼起。
systemctl enable kubelet
十、安装haproxy和keepalived(三个master节点)
1.在三台master上安装keepalived
####
yum install -y keepalived
cat > /etc/keepalived/keepalived.conf <<EOF
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
#vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 {
state BACKUP
interface enp0s3
virtual_router_id 51
priority 70
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.56.140
}
}
EOF
####启动
systemctl enable keepalived && systemctl start keepalived
###查看虚拟ip是不是在master1上,关闭master1的keepalived服务,测试下vip是否漂移到其他机器
ip a
2.安装haproxy(三个master节点)
###三台master上配置一样
yum install -y haproxy
cat > /etc/haproxy/haproxy.cfg <<EOF
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
defaults
mode tcp
log global
retries 3
timeout connect 10s
timeout client 1m
timeout server 1m
frontend kubernetes
bind *:8443
mode tcp
default_backend kubernetes-master
backend kubernetes-master
balance roundrobin
server master1 192.168.56.101:6443 check maxconn 2000
server master2 192.168.56.102:6443 check maxconn 2000
server master3 192.168.56.103:6443 check maxconn 2000
EOF
####启动
systemctl enable haproxy && systemctl start haproxy
十、部署master1节点(k8s-master-1, 192.168.56.101),要求最少要2核心
- ####Master节点执行初始化,通过image-repository参数修改指定从阿里云镜像或私库下载相关镜像
有互联网环境下载时:
####k8s-master-1结点初始化
kubeadm init \
--control-plane-endpoint "192.168.56.140:8443" \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.21.2 \
--pod-network-cidr=10.244.0.0/16 \
--upload-certs
要从私库下载时:
####k8s-master-1结点初始化
kubeadm init \
--control-plane-endpoint "192.168.56.140:8443" \
--image-repository 192.168.56.1:5000/google_containers \
--kubernetes-version v1.21.2 \
--pod-network-cidr=10.244.0.0/16 \
--upload-certs
如果你有互联网环境,直接使用有互联网环境的命令即可,互联网的命令指定了从阿里云下载镜像。
我使用的是私库下载的模式,因为公司的正式环境基本上不让上外网,初始化过程如下---->>>>
[root@k8s-master-1 ~]# kubeadm init \
> --control-plane-endpoint "192.168.56.140:8443" \
> --image-repository 192.168.56.1:5000/google_containers \
> --kubernetes-version v1.21.2 \
> --pod-network-cidr=10.244.0.0/16 \
> --upload-certs
[init] Using Kubernetes version: v1.21.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.56.101 192.168.56.140]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master-1 localhost] and IPs [192.168.56.101 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master-1 localhost] and IPs [192.168.56.101 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 14.529641 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.21" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
73b0b6328e0193557d46ff6cb0f01e2c44967fe4e2bacde4d9ccee1baed6d011
[mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: ir8tfi.ezn32y97tympnrtn
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join 192.168.56.140:8443 --token ir8tfi.ezn32y97tympnrtn \
--discovery-token-ca-cert-hash sha256:4f42525ddf9535fb69a5bb4a830d39e190d9c5bb3b0760e279c27460e7d79e19 \
--control-plane --certificate-key 73b0b6328e0193557d46ff6cb0f01e2c44967fe4e2bacde4d9ccee1baed6d011
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.56.140:8443 --token ir8tfi.ezn32y97tympnrtn \
--discovery-token-ca-cert-hash sha256:4f42525ddf9535fb69a5bb4a830d39e190d9c5bb3b0760e279c27460e7d79e19
配置kubectl工具:
####配置kubectl工具
####机器上的用户要使用 kubectl来 管理集群操作集群,需要做如下配置
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
十一、部署k8s-master-2和k8s-master-3
####在k8s-master-2和k8s-master-3上执行(在成功初始化k8s-master-1的结果中找命令)
kubeadm join 192.168.56.140:8443 --token ir8tfi.ezn32y97tympnrtn \
--discovery-token-ca-cert-hash sha256:4f42525ddf9535fb69a5bb4a830d39e190d9c5bb3b0760e279c27460e7d79e19 \
--control-plane --certificate-key 73b0b6328e0193557d46ff6cb0f01e2c44967fe4e2bacde4d9ccee1baed6d011
####同样,在k8s-master-2和k8s-master-3上配置kubectl工具
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
十一、执行集群检查
kubectl get nodes
####结果如下:没有安装网络插件,因此都是NotReady
[root@k8s-master-1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master-1 NotReady control-plane,master 6m52s v1.21.2
k8s-master-2 NotReady control-plane,master 91s v1.21.2
k8s-master-3 NotReady control-plane,master 46s v1.21.2
十二、部署网络插件(k8s-master节点, 192.168.56.101)
网络插件可以选择calico网络,也可以选择flannel,这里我们选用calico
- ####下载calico.yaml
wget https://docs.projectcalico.org/v3.19/manifests/calico.yaml
- ####修改calico.yaml,将所有image下加上一行imagePullPolicy: IfNotPresent,使得k8s在本地有了镜像后不再从网上下载。
- ####calico在多网络接口时自动检测到错误的网络接口,导致网络无法连通,通过指定网络接口(网卡名)解决问题
在calico.yaml中的
- name: CLUSTER_TYPE
value: "k8s,bgp"
下面增加两行
- name: IP_AUTODETECTION_METHOD
value: "interface=enp0s3"
- ####执行kubectl apply -f calico.yaml进行calica.yaml的部署
kubectl apply -f calica.yaml
报了一个warning提示,根据提示后将calica.yaml中的policy/v1beta1修改为policy/v1
修改成
然后重新应用
kubectl apply -f calica.yaml
结果如下:
5. ####私网没外网环境下镜像手工下载准备
如果你是有互联网,执行上面的那一步,顺利的话就能直接成功了。
但我没有外网,因此需要先在有外网的docker环境下下载好以下的4个镜像
,然后将保存到本地的镜像上传到内网的docker服务器,执行docker load -i进行镜像的导入,将镜像导入所有的节点
我也把文件放在百度盘备用:
链接:https://pan.baidu.com/s/1l75HAnjQhbUVfvG-O6tc2w
提取码:cxe1
#拉取
docker pull docker.io/calico/cni:v3.19.4
#保存到本地
docker save docker.io/calico/cni:v3.19.4 -o cni_v3.19.4.tar.gz
#重新导入
docker load -i cni_v3.19.4.tar.gz
docker.io/calico/pod2daemon-flexvol:v3.19.4
docker save docker.io/calico/pod2daemon-flexvol:v3.19.4 -o flexvol_v3.19.4.tar.gz
docker load -i flexvol_v3.19.4.tar.gz
docker pull docker.io/calico/node:v3.19.4
docker save docker.io/calico/node:v3.19.4 node_v3.19.4.tar.gz
docker load -i node_v3.19.4.tar.gz
docker pull docker.io/calico/kube-controllers:v3.19.4
docker save docker.io/calico/kube-controllers:v3.19.4 -o controllers_v3.19.4.tar.gz
docker load -i controllers_v3.19.4.tar.gz
重新导入镜像后集群成功部署
[root@k8s-master-1 testuser]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-master-1 Ready control-plane,master 60m v1.21.2 192.168.56.101 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://20.10.7
k8s-master-2 Ready control-plane,master 55m v1.21.2 192.168.56.102 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://20.10.7
k8s-master-3 Ready control-plane,master 54m v1.21.2 192.168.56.103 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://20.10.7
kubectl get pods --all-namespaces查看pod的状态
[root@k8s-master-1 testuser]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-7cc8dd57d9-vtxxs 1/1 Running 0 5m14s
kube-system calico-node-7znz5 1/1 Running 0 5m14s
kube-system calico-node-njtx6 1/1 Running 0 5m14s
kube-system calico-node-z7cfg 1/1 Running 0 5m14s
kube-system coredns-5c9594667d-2tbr2 1/1 Running 0 8m48s
kube-system coredns-5c9594667d-trjbv 1/1 Running 0 8m48s
kube-system etcd-k8s-master-1 1/1 Running 0 8m56s
kube-system etcd-k8s-master-2 1/1 Running 0 8m20s
kube-system etcd-k8s-master-3 1/1 Running 0 8m7s
kube-system kube-apiserver-k8s-master-1 1/1 Running 0 8m56s
kube-system kube-apiserver-k8s-master-2 1/1 Running 0 8m21s
kube-system kube-apiserver-k8s-master-3 1/1 Running 0 7m52s
kube-system kube-controller-manager-k8s-master-1 1/1 Running 1 8m56s
kube-system kube-controller-manager-k8s-master-2 1/1 Running 0 8m21s
kube-system kube-controller-manager-k8s-master-3 1/1 Running 0 7m59s
kube-system kube-proxy-8cd27 1/1 Running 0 7m44s
kube-system kube-proxy-krtj4 1/1 Running 0 8m49s
kube-system kube-proxy-mv272 1/1 Running 0 8m22s
kube-system kube-scheduler-k8s-master-1 1/1 Running 1 8m56s
kube-system kube-scheduler-k8s-master-2 1/1 Running 0 8m21s
kube-system kube-scheduler-k8s-master-3 1/1 Running 0 8m7s
十三、部署k8s-node-1和k8s-node-2
部署node结点就简单了,直接执行加入命令就可
kubeadm join 192.168.56.140:8443 --token ir8tfi.ezn32y97tympnrtn \
--discovery-token-ca-cert-hash sha256:4f42525ddf9535fb69a5bb4a830d39e190d9c5bb3b0760e279c27460e7d79e19
至此,高可用三master节点的集群搭建完成.