kubernetes1.16 K8S高可用部署--三主三从

最新推荐文章于 2024-08-09 12:01:49 发布
最新推荐文章于 2024-08-09 12:01:49 发布
阅读量2.9k 收藏 14
点赞数
分类专栏： 容器 文章标签： kubernetes高可用 ansible部署kubernetes kubernetes三主三从 kubernetes节点去污
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_36783142/article/details/102674688
版权

目录


一.环境说明

一.环境说明

操作系统：centos7
kubernetes：16.0
docker：18.06

主机名IP地址类型
master-1192.168.1.181masters
master-2192.168.1.182masters
master-3192.168.1.183masters
node-1192.168.1.184nodes
node-2192.168.1.185nodes
node-3192.168.1.186nodes
vip192.168.1.180vips

以下操作若无特别说明，默认在master-1上执行

二.生成秘钥，使所有主机能够免密码登录

[root@localhost ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:2iC7vtK7x1XEdXhvQMhHTAJ/4XfP1WyFGfID5siGKA0 root@localhost.localdomain
The key's randomart image is:
+---[RSA 2048]----+
|    E    ..+*BB+.|
|     o . oo=+BBoo|
|    . o ..+ oo=oB|
|     .   ..  . =*|
|    . . S.     .o|
|     o +.        |
|   ......        |
|  . ..o          |
|   o*=           |
+----[SHA256]-----+
## 发送公钥到各个节点，实现免密码登陆
[root@localhost ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.1.181:
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.1.181 (192.168.1.181)' can't be established.
ECDSA key fingerprint is SHA256:ouDBNSO0P+fR3kL+QIpSw6A6uHZa4JF+REA2+30yAgE.
ECDSA key fingerprint is MD5:db:eb:bb:8c:b2:fb:6e:70:49:19:87:99:d8:58:3a:78.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.1.181's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '192.168.1.181'"
and check to make sure that only the key(s) you wanted were added.

[root@localhost ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.1.182:
[root@localhost ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.1.183:
[root@localhost ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.1.184:
[root@localhost ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.1.185:
[root@localhost ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub 192.168.1.186:

三.添加阿里云yum源，并安装ansible自动化管理工具

[root@localhost ~]# curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo &&  yum -y install ansible

四.生成ansible的hosts文件

[root@localhost ~]# cat  >/etc/ansible/hosts  <<EOF
[masters]
master-1 ansible_ssh_host="192.168.1.181"
master-2 ansible_ssh_host="192.168.1.182"
master-3 ansible_ssh_host="192.168.1.183      "

[nodes]
node-1 ansible_ssh_host="192.168.1.184"
node-2 ansible_ssh_host="192.168.1.185"
node-3 ansible_ssh_host="192.168.1.186"
EOF

五.关闭防火墙、selinux

[root@master-1 ~]# ansible all -m  shell -a "setenforce 0 && systemctl stop firewalld && systemctl disable firewalld&& sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux"

六.修改主机名

[root@localhost ~]# ansible master-1 -m shell -a "hostnamectl set-hostname  master-1"
[root@localhost ~]# ansible master-2 -m shell -a "hostnamectl set-hostname  master-2"
[root@localhost ~]# ansible master-3 -m shell -a "hostnamectl set-hostname  master-3"
[root@localhost ~]# ansible node-1 -m shell -a "hostnamectl set-hostname  node-1  "
[root@localhost ~]# ansible node-2 -m shell -a "hostnamectl set-hostname  node-2 "
[root@localhost ~]# ansible node-3 -m shell -a "hostnamectl set-hostname  node-3"

七.添加本地/etc/hosts/文件

[root@master-1 ~]# ansible all -m shell -a "echo '192.168.1.180 vip'>> /etc/hosts  "
[root@master-1 ~]# ansible all -m shell -a "echo '192.168.1.181 master-1'>> /etc/hosts  "
[root@master-1 ~]# ansible all -m shell -a "echo '192.168.1.182 master-1'>> /etc/hosts  "
[root@master-1 ~]# ansible all -m shell -a "echo '192.168.1.183 master-1'>> /etc/hosts  "
[root@master-1 ~]# ansible all -m shell -a "echo '192.168.1.186 node-3'>> /etc/hosts  "
[root@master-1 ~]# ansible all -m shell -a "echo '192.168.1.184 node-2'>> /etc/hosts  "
[root@master-1 ~]# ansible all -m shell -a "echo '192.168.1.185 node-1'>> /etc/hosts  "
##查看/etc/hosts文件是否正确
[root@master-1 ~]# ansible all -m shell -a "cat /etc/hosts  "

八.同步所有主机的yum源

## 安装基础命令
[root@master-1 ~]# ansible all -m  shell -a  "yum install yum-utils device-mapper-persistent-data lvm2 vim wget yum-utils lrzsz lsof -y "
## 配置docker、kubernetes阿里云yum源
[root@master-1 ~]#  curl -o /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@master-1~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg 
EOF
[root@master-1 ~]#  curl -o /tmp/rpm-package-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
[root@master-1 ~]# ansible all -m copy -a "src=/tmp/rpm-package-key.gpg dest=/tmp/rpm-package-key.gpg"
[root@master-1 ~]# ansible all -m copy -a "src=/etc/yum.repos.d/docker-ce.repo dest=/etc/yum.repos.d/docker-ce.repo"
[root@master-1 ~]# ansible all -m copy -a "src=/etc/yum.repos.d/kubernetes.repo dest=/etc/yum.repos.d/kubernetes.repo"

九.修改内核参数，并关闭swap分区

[root@master-1 ~]# 
[root@master-1 ~]# cat <<EOF >  /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
[root@master-1 ~]# ansible all -m copy -a "src=/etc/sysctl.d/k8s.conf dest=/etc/sysctl.d/k8s.conf"
[root@master-1 ~]# ansible all -m  shell -a  "modprobe br_netfilter && sysctl -p /etc/sysctl.d/k8s.conf"
## 关闭swap分区
[root@master-1 ~]# ansible all -m  shell -a  "sed -i '/swap/ s/^/#/' /etc/fstab && swapoff -a"

十.配置docker加速镜像地址

[root@master-1 ~]# ansible all -m  shell -a  "mkdir -p /etc/docker &&mkdir -p /etc/systemd/system/docker.service.d"
[root@master-1 ~]# cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ],
  "registry-mirrors": ["https://v2ltjwbg.mirror.aliyuncs.com"]

}
EOF
[root@master-1 ~]# ansible all -m copy -a "src=/etc/docker/daemon.json dest=/etc/docker/daemon.json"

十一.安装keepalived、kubeadm-1.16.0、kubectl-1.16.0、kubelet-1.16.0、docker-ce-18.06.1.ce-3.el7

[root@master-1 ~]# vip="192.168.1.180"
[root@master-1 ~]# masters="masters"
[root@master-1 ~]# ansible ${masters} -m  shell -a  "yum -y install keepalived kubeadm-1.16.0 kubectl-1.16.0 kubelet-1.16.0 docker-ce-18.06.1.ce-3.el7 "
[root@master-1 ~]# ansible ${masters} -m  shell -a  "mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak"
[root@master-1 ~]# virtual_router_id="140"

[root@master-1 ~]# cat <<EOF > /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   router_id HOSTNAME                      #主调度器的主机名


}

vrrp_instance VI_1 {
    state BACKUP                          
    interface ens33
    virtual_router_id ${virtual_router_id}              
    nopreempt                             
    priority priority_mun                         
    advert_int 1
    authentication {
        auth_type PASS                     
        auth_pass 123456                 
    }
    virtual_ipaddress {
        ${vip}                            #VIP地址声明
    }
}
EOF
[root@master-1 ~]# ansible ${masters} -m copy -a "src=/etc/keepalived/keepalived.conf dest=/etc/keepalived/keepalived.conf"
[root@master-1 ~]# sed -i 's/BACKUP/MASTER/g' /etc/keepalived/keepalived.conf
[root@master-1 ~]# priority_mun="90"
[root@master-1 ~]# ip="192.168.1.181"
[root@master-1 ~]# hostname=$(ssh -p22 ${ip} "hostname")
[root@master-1 ~]# ssh -p22 ${ip} "sed -i 's/priority_mun/${priority_mun}/g' /etc/keepalived/keepalived.conf"
[root@master-1 ~]# ssh -p22 ${ip} "sed -i 's/HOSTNAME/${hostname}/g' /etc/keepalived/keepalived.conf"
[root@master-1 ~]# priority_mun=$(( $priority_mun + 15 ))
[root@master-1 ~]# ip="192.168.1.182"
[root@master-1 ~]# hostname=$(ssh -p22 ${ip} "hostname")
[root@master-1 ~]# ssh -p22 ${ip} "sed -i 's/priority_mun/${priority_mun}/g' /etc/keepalived/keepalived.conf"
[root@master-1 ~]# ssh -p22 ${ip} "sed -i 's/HOSTNAME/${hostname}/g' /etc/keepalived/keepalived.conf"
[root@master-1 ~]# priority_mun=$(( $priority_mun + 15 ))
[root@master-1 ~]# ip="192.168.1.183"
[root@master-1 ~]# hostname=$(ssh -p22 ${ip} "hostname")
[root@master-1 ~]# ssh -p22 ${ip} "sed -i 's/priority_mun/${priority_mun}/g' /etc/keepalived/keepalived.conf"
[root@master-1 ~]# ssh -p22 ${ip} "sed -i 's/HOSTNAME/${hostname}/g' /etc/keepalived/keepalived.conf"
[root@master-1 ~]# ansible masters -m  shell -a  "systemctl enable  keepalived && systemctl start   keepalived"
[root@master-1 ~]# ansible all -m  shell -a  "systemctl enable docker kubelet && systemctl start  docker kubelet"

十二.使用kubeadm初始化kubernetes集群


[root@master-1 ~]# cat <<EOF > /etc/kubernetes/kubeadm.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
  certSANs:
  - "${vip}"  #请求改为你的vip地址
controlPlaneEndpoint: "${vip}:6443"  #请求改为你的vip地址
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
networking:
  dnsDomain: cluster.local
  podSubnet: "10.244.0.0/16"
  serviceSubnet: 10.96.0.0/12
EOF
##  初始化kubernetes集群
[root@master-1 ~]# kubeadm init --config  /etc/kubernetes/kubeadm.yaml       
W1021 23:26:53.008747   22742 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get https://dl.k8s.io/release/stable.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1021 23:26:53.009179   22742 version.go:102] falling back to the local client version: v1.16.0
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.181 192.168.1.180 192.168.1.180]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master-1 localhost] and IPs [192.168.1.181 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master-1 localhost] and IPs [192.168.1.181 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 38.505409 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master-1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 9fsbcu.dzf3ac971y86it9p
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities 
and service account keys on each node and then running the following as root:

  kubeadm join 192.168.1.180:6443 --token 9fsbcu.dzf3ac971y86it9p \
    --discovery-token-ca-cert-hash sha256:736da2c4da8d3df65057740f3c51808e7613d5670cd0b4c63bbbfffaa56f3a0b \
    --control-plane       

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.180:6443 --token 9fsbcu.dzf3ac971y86it9p \
    --discovery-token-ca-cert-hash sha256:736da2c4da8d3df65057740f3c51808e7613d5670cd0b4c63bbbfffaa56f3a0b 
##这里我们看到的集群ip地址为：192.168.1.180，这是keepalived产生的虚拟ip


[root@master-1 ~]#  mkdir -p $HOME/.kube
[root@master-1 ~]#  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master-1 ~]#  sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 查看集群，状态为NotReady，这是因为我们没有添加网络插件，这里选择flannel网络，后面再添加； 
[root@master-1 ~]# kubectl get node 
NAME       STATUS     ROLES    AGE     VERSION
master-1   NotReady   master   3m58s   v1.16.0

十三.其他master加入集群

## 输出秘钥
[root@master-1 ~]# ssh 192.168.1.182 "mkdir -p /etc/kubernetes/pki/etcd/"
[root@master-1 ~]# ssh 192.168.1.183 "mkdir -p /etc/kubernetes/pki/etcd/"
## master-2
[root@master-1 ~]# scp -r /etc/kubernetes/admin.conf 192.168.1.182:/etc/kubernetes/
[root@master-1 ~]# scp -r /etc/kubernetes/pki/ca* 192.168.1.182:/etc/kubernetes/pki/
[root@master-1 ~]# scp -r /etc/kubernetes/pki/sa* 192.168.1.182:/etc/kubernetes/pki/
[root@master-1 ~]# scp -r /etc/kubernetes/pki/front* 192.168.1.182:/etc/kubernetes/pki/
[root@master-1 ~]# scp -r /etc/kubernetes/pki/etcd/ca* 192.168.1.182:/etc/kubernetes/pki/etcd/
[root@master-1 ~]# scp -r /etc/kubernetes/pki/etcd/ser* 192.168.1.182:/etc/kubernetes/pki/etcd/
## master-3
[root@master-1 ~]# scp -r /etc/kubernetes/admin.conf 192.168.1.183:/etc/kubernetes/
[root@master-1 ~]# scp -r /etc/kubernetes/pki/ca* 192.168.1.183:/etc/kubernetes/pki/
[root@master-1 ~]# scp -r /etc/kubernetes/pki/sa* 192.168.1.183:/etc/kubernetes/pki/
[root@master-1 ~]# scp -r /etc/kubernetes/pki/front* 192.168.1.183:/etc/kubernetes/pki/
[root@master-1 ~]# scp -r /etc/kubernetes/pki/etcd/ca* 192.168.1.183:/etc/kubernetes/pki/etcd/
[root@master-1 ~]# scp -r /etc/kubernetes/pki/etcd/ser* 192.168.1.183:/etc/kubernetes/pki/etcd/
## 加入master集群
###获取token值
[root@master-1 ~]# kubeadm token list | awk '{print $1}' | sed "1d"
9fsbcu.dzf3ac971y86it9p
[root@master-1 ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
sha256:736da2c4da8d3df65057740f3c51808e7613d5670cd0b4c63bbbfffaa56f3a0b

## master-2加入到masters
[root@master-2 ~]#  kubeadm join 192.168.1.180:6443 --token 9fsbcu.dzf3ac971y86it9p \
    --discovery-token-ca-cert-hash sha256:736da2c4da8d3df65057740f3c51808e7613d5670cd0b4c63bbbfffaa56f3a0b \
    --control-plane   
[root@master-2 ~]#  mkdir -p $HOME/.kube
[root@master-2 ~]#  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master-2 ~]#  sudo chown $(id -u):$(id -g) $HOME/.kube/config
## master-3加入到masters
[root@master-3 ~]#  kubeadm join 192.168.1.180:6443 --token 9fsbcu.dzf3ac971y86it9p \
    --discovery-token-ca-cert-hash sha256:736da2c4da8d3df65057740f3c51808e7613d5670cd0b4c63bbbfffaa56f3a0b \
    --control-plane   
[root@master-3 ~]#  mkdir -p $HOME/.kube
[root@master-3 ~]#  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master-3 ~]#  sudo chown $(id -u):$(id -g) $HOME/.kube/config

十四.node节点加入集群

[root@node-1 ~]# kubeadm join 192.168.1.180:6443 --token 9fsbcu.dzf3ac971y86it9p \
    --discovery-token-ca-cert-hash sha256:736da2c4da8d3df65057740f3c51808e7613d5670cd0b4c63bbbfffaa56f3a0b 
[root@node-2 ~]# kubeadm join 192.168.1.180:6443 --token 9fsbcu.dzf3ac971y86it9p \
    --discovery-token-ca-cert-hash sha256:736da2c4da8d3df65057740f3c51808e7613d5670cd0b4c63bbbfffaa56f3a0b 
[root@node-3 ~]# kubeadm join 192.168.1.180:6443 --token 9fsbcu.dzf3ac971y86it9p \
    --discovery-token-ca-cert-hash sha256:736da2c4da8d3df65057740f3c51808e7613d5670cd0b4c63bbbfffaa56f3a0b

十五.添加flannel网络

[root@master-1 ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
## 查看集群kubectl get node 
[root@master-1 ~]# kubectl get node
NAME       STATUS     ROLES    AGE     VERSION
master-1   NotReady   master   17m     v1.16.0
master-2   NotReady   master   8m40s   v1.16.0
master-3   NotReady   master   6m19s   v1.16.0
node-1     NotReady   <none>   5m57s   v1.16.0
node-2     NotReady   <none>   5m43s   v1.16.0
node-3     NotReady   <none>   5m51s   v1.16.0

十六.master去污点

## 这里发现coredns的状态为Pending，并且kube-flannel-ds-amd64的状态也不是为running
[root@master-1 ~]# kubectl get pod -n kube-system                               
NAME                               READY   STATUS     RESTARTS   AGE
coredns-67c766df46-hn9fp           0/1     Pending    0          17m
coredns-67c766df46-t99xw           0/1     Pending    0          17m
etcd-master-1                      1/1     Running    2          16m
etcd-master-2                      1/1     Running    0          9m10s
etcd-master-3                      1/1     Running    0          6m48s
kube-apiserver-master-1            1/1     Running    1          16m
kube-apiserver-master-2            1/1     Running    0          9m10s
kube-apiserver-master-3            1/1     Running    0          6m48s
kube-controller-manager-master-1   1/1     Running    2          16m
kube-controller-manager-master-2   1/1     Running    1          9m10s
kube-controller-manager-master-3   1/1     Running    0          6m48s
kube-flannel-ds-amd64-8df8p        0/1     Init:0/1   0          6s
kube-flannel-ds-amd64-8jnmv        0/1     Init:0/1   0          6s
kube-flannel-ds-amd64-ctpm6        0/1     Init:0/1   0          6s
kube-flannel-ds-amd64-g64k5        0/1     Init:0/1   0          6s
kube-flannel-ds-amd64-qxqdc        0/1     Init:0/1   0          6s
kube-flannel-ds-amd64-t6cxp        0/1     Init:0/1   0          6s
kube-proxy-78xp8                   1/1     Running    0          6m13s
kube-proxy-8qqck                   1/1     Running    0          6m21s
kube-proxy-hmznl                   1/1     Running    0          6m27s
kube-proxy-lwtsq                   1/1     Running    1          17m
kube-proxy-rdktz                   1/1     Running    0          6m49s
kube-proxy-wqq88                   1/1     Running    0          9m10s
kube-scheduler-master-1            1/1     Running    2          16m
kube-scheduler-master-2            1/1     Running    1          9m10s
kube-scheduler-master-3            1/1     Running    0          6m48s

## 使用kubectl describe pod xxxx -n kube-system命令排查，发现had taints that the pod didn't tolerate.

[root@master-1 ~]# kubectl describe pod coredns-67c766df46-hn9fp  -n kube-system | tail -n 10
  Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/4 nodes are available: 4 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/5 nodes are available: 5 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/6 nodes are available: 6 node(s) had taints that the pod didn't tolerate.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/6 nodes are available: 6 node(s) had taints that the pod didn't tolerate.
  


## 发现master节点是不可调度的。这是因为kubernetes出于安全考虑默认情况下无法在master节点上部署pod
[root@master-1 ~]# kubectl get no -o yaml | grep taint -A 5
    taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
    - effect: NoSchedule
      key: node.kubernetes.io/not-ready
  status:
--
    taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
    - effect: NoSchedule
      key: node.kubernetes.io/not-ready
  status:
--
    taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
    - effect: NoSchedule
      key: node.kubernetes.io/not-ready
  status:
--
    taints:
    - effect: NoSchedule
      key: node.kubernetes.io/not-ready
  status:
    addresses:
    - address: 192.168.1.184
--
    taints:
    - effect: NoSchedule
      key: node.kubernetes.io/not-ready
  status:
    addresses:
    - address: 192.168.1.185
--
    taints:
    - effect: NoSchedule
      key: node.kubernetes.io/not-ready
  status:
    addresses:
    - address: 192.168.1.186
## 去污操作
[root@master-1 ~]# kubectl taint nodes --all node-role.kubernetes.io/master-

## 稍等一会，再次查看集群信息，STATUS状态为Ready则表示集群环境搭建成功
[root@master-1 ~]# kubectl get node
NAME       STATUS     ROLES    AGE     VERSION
master-1   Ready   master   17m     v1.16.0
master-2   Ready   master   8m40s   v1.16.0
master-3   Ready   master   6m19s   v1.16.0
node-1     Ready   <none>   5m57s   v1.16.0
node-2     Ready   <none>   5m43s   v1.16.0
node-3     Ready   <none>   5m51s   v1.16.0

flannel镜像pull失败

[root@master-1 ~]# kubectl describe pod kube-flannel-ds-amd64-kwjxv -n kube-system | tail -n 10
  Normal   Scheduled  <unknown>             default-scheduler  Successfully assigned kube-system/kube-flannel-ds-amd64-kwjxv to master-1
  Warning  Failed     23m                   kubelet, master-1  Failed to pull image "quay.io/coreos/flannel:v0.11.0-amd64": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/coreos/flannel/manifests/v0.11.0-amd64: read tcp 192.168.1.181:33496->54.225.149.151:443: read: connection timed out
  Normal   Pulling    22m (x4 over 36m)     kubelet, master-1  Pulling image "quay.io/coreos/flannel:v0.11.0-amd64"
  Warning  Failed     20m (x4 over 31m)     kubelet, master-1  Error: ErrImagePull
  Warning  Failed     9m41s (x5 over 31m)   kubelet, master-1  Failed to pull image "quay.io/coreos/flannel:v0.11.0-amd64": rpc error: code = Unknown desc = context canceled
  Normal   BackOff    5m26s (x42 over 31m)  kubelet, master-1  Back-off pulling image "quay.io/coreos/flannel:v0.11.0-amd64"
  Warning  Failed     36s (x62 over 31m)    kubelet, master-1  Error: ImagePullBackOff

## 手动pull镜像
[root@master-1 ~]# docker pull quay.io/coreos/flannel:v0.11.0-amd64
v0.11.0-amd64: Pulling from coreos/flannel
cd784148e348: Pull complete 
04ac94e9255c: Pull complete 
e10b013543eb: Pull complete 
005e31e443b1: Pull complete 
74f794f05817: Pull complete 
Digest: sha256:7806805c93b20a168d0bbbd25c6a213f00ac58a511c47e8fa6409543528a204e
Status: Downloaded newer image for quay.io/coreos/flannel:v0.11.0-amd64
再次查看pod
[root@master-1 ~]# kubectl get po --all-namespaces               
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-67c766df46-qbvp4           1/1     Running   0          67m
kube-system   coredns-67c766df46-qrdhz           1/1     Running   0          67m
kube-system   etcd-master-1                      1/1     Running   0          66m
kube-system   etcd-master-2                      1/1     Running   0          65m
kube-system   etcd-master-3                      1/1     Running   0          58m
kube-system   kube-apiserver-master-1            1/1     Running   0          67m
kube-system   kube-apiserver-master-2            1/1     Running   0          65m
kube-system   kube-apiserver-master-3            1/1     Running   0          58m
kube-system   kube-controller-manager-master-1   1/1     Running   2          66m
kube-system   kube-controller-manager-master-2   1/1     Running   0          65m
kube-system   kube-controller-manager-master-3   1/1     Running   1          58m
kube-system   kube-flannel-ds-amd64-645cv        1/1     Running   0          55m
kube-system   kube-flannel-ds-amd64-92cdd        1/1     Running   0          55m
kube-system   kube-flannel-ds-amd64-gmsjj        1/1     Running   0          55m
kube-system   kube-flannel-ds-amd64-kwjxv        1/1     Running   0          55m
kube-system   kube-flannel-ds-amd64-lft5p        1/1     Running   0          55m
kube-system   kube-flannel-ds-amd64-lxh2b        1/1     Running   0          55m
kube-system   kube-proxy-7qwvv                   1/1     Running   0          58m
kube-system   kube-proxy-8sbl9                   1/1     Running   0          67m
kube-system   kube-proxy-nn5ms                   1/1     Running   0          58m
kube-system   kube-proxy-vbmhc                   1/1     Running   0          65m
kube-system   kube-proxy-x9rr9                   1/1     Running   0          57m
kube-system   kube-proxy-xtt2p                   1/1     Running   0          57m
kube-system   kube-scheduler-master-1            1/1     Running   1          66m
kube-system   kube-scheduler-master-2            1/1     Running   2          65m
kube-system   kube-scheduler-master-3            1/1     Running   0          58m
小~苏
关注
专栏目录
【Flink on k8s】- 1 - Flink Kubernetes Operator 简介 2023
欢迎来到我的博客，一起探索代码里的世界！
03-22 822
Flink Kubernetes Operator 可以与 Kubernetes 的其他功能和工具集成，可以帮助用户快速定位和解决问题。Flink Kubernetes Operator可以在Kubernetes上实现高可用性，以确保 Flink 集群的稳定性和可靠性。Flink Kubernetes Operator 可以快速地部署 Flink 集群，而无需手动配置和管理集群。Flink Kubernetes Operator可以将Flink集群与其他应用程序隔离，以避免资源争用和干扰。
Kubernetes集群搭建(高可用)附全自动脚本
以梦为马，不负韶华
04-01 2075
clustercluster是 计算、存储和网络资源的集合，k8s利用这些资源运行各种基于容器的应用。mastermaster是cluster的大脑，他的主要职责是调度，即决定将应用放在那里运行。master运行linux操作系统，可以是物理机或者虚拟机。为了实现高可用，可以运行多个master。nodenode的职责是运行容器应用。node由master管理，node负责监控并汇报容器的状态，同时根据master的要求管理容器的生命周期。node运行在linux的操作系统上，可以是物理机或者是虚拟机。
k8s高可用集群部署
weixin_68562945的博客
07-03 621
对master节点做高可用当高可用集群当中任意节点发生瘫痪会有其他master节点继续提供服务准备三台master节点 （server1 server2 server3）在实现k8s集群高可用时时通过负载均衡层来进行的 所以在实施的时候也要对负载均衡层来实现高可用 （当负载均衡层崩溃时，会有其他的负载均衡层来进行接管）先准备两台机器用来做高可用套件（高可用加负载均衡层）在两台机器上安装 haproxy.x86_64 0:1.5.18-9.el7_9.1编辑文件/etc/haproxy/haproxy.
k8s高可用部署
wuzhicong_1的博客
04-23 234
多主多从：多个Master节点和多台Node节点，使用VIP漂移，，用于生产环境。使用kubeadm，一个用于快速搭建kubernetes集群的工具。
Kubernetes搭建高可用集群(keepalived+haproxy)
最新发布
qq_40914472的博客
08-09 1540
Kubernetes搭建高可用集群(keepalived+haproxy)
【kubernetes】k8s高可用集群搭建（三主三从）
2302_77582029的博客
08-15 7421
【kubernetes】k8s高可用集群搭建（三主三从）
详解 K8S 高可用部署
油墨香^-^的博客
12-05 2176
一、前言二、基础环境部署1）前期准备（所有节点）2）安装容器 docker（所有节点）3）配置 k8s yum 源（所有节点）4）将 sandbox_image 镜像源设置为阿里云 google_containers 镜像源（所有节点）5）配置 containerd cgroup 驱动程序 systemd（所有节点）6）开始安装 kubeadm，kubelet 和 kubectl（master 节点）7）使用 kubeadm 初始化集群（master 节点）8）安装 Pod 网络插件（CNI：Contain
k8s之高可用集群部署
weixin_43757555的博客
09-03 332
初始化系统 所有服务器初始化节点，在之前部署k8s有提到，这里不做赘述，请参考 https://blog.csdn.net/weixin_43757555/article/details/108142265 初始化完成后关闭 NUMA cp /etc/default/grub{,.bak} vim /etc/default/grub # 在 GRUB_CMDLINE_LINUX 一行添加 `numa=off` 参数，如下所示： < GRUB_CMDLINE_LINUX="crashkernel=aut
使用kubeadm搭建高可用集群-k8s相关组件及1.16版本的安装部署
weixin_43805705的博客
11-24 835
k8s相关组件及1.16版本的安装部署
kubernetes高可用集群搭建---kubeadm方式（基于1.16.1）
360扣扣管家的博客
10-14 1507
环境准备 安装版本 CentOS Linux release 7.6.1810 (Core) Docker 18.09.7 kubeadm 1.16.1 准备工作 这里我使用了5台centos7的虚拟机，3台master，2台worker节点，相关配置如下： 主机名 IP地址 角色 cpu Memory k8s-master-01 128.0.1.190 master 4 8G...
kubeadm部署kubernetes（k8s）
yrx420909的博客
04-21 1400
1. Kubernetes简介 官方文档：https://kubernetes.io/docs/home/ 在Docker 作为高级容器引擎快速发展的同时，在Google内部，容器技术已经应用了很多年，Borg系统运行管理着成千上万的容器应用。 Kubernetes项目来源于Borg，可以说是集结了Borg设计思想的精华，并且吸收了Borg系统中的经验和教训。 Kubernetes对计算资源进行了...
部署一套完整的K8s高可用集群（二进制-V1.18）.zip
07-11
从github下载发行版的二进制包，手动部署每个组件，组成Kubernetes集群。 Kubeadm降低部署门槛，但屏蔽了很多细节，遇到问题很难排查。如果想更容易可控，推荐使用二进 制包部署Kubernetes集群，虽然手动部署麻烦点，期间可以学习很多工作原理，也利于后期维护。
k8s高可用部署方案
weixin_44770684的博客
04-05 1479
k8s高可用部署方案
K8s高可用集群部署
weixin_30600197的博客
06-29 139
https://github.com/sky-daiji/salt-k8s-ha-v2 转载于:https://www.cnblogs.com/durenniu/p/11107532.html
K8S高可用集群部署
weixin_44772835的博客
12-06 1342
准备环境 软件环境：软件版本操作系统CentOS7.8_x64 (mini)Docker20-ceKubernetes1.25服务器整体规划：角色IP其他单装组件k8s-master1192.168.1.71docker，etcd，nginx，keepalivedk8s-master2192.168.1.72docker，...
3.6万字详解 K8S 高可用部署，超详细！
weixin_45727359的博客
08-30 730
一、前言二、基础环境部署1）前期准备（所有节点）2）安装容器 docker（所有节点）3）配置 k8s yum 源（所有节点）4）将 sandbox_image 镜像源设置为阿里云 google_containers 镜像源（所有节点）5）配置 containerd cgroup 驱动程序 systemd（所有节点）6）开始安装 kubeadm，kubelet 和 kubectl（master 节...
k8s-v1.24.2高可用部署
mylove_0的博客
08-31 4862
k8s-v1.24.2高可用部署
Kubernetes 18 （k8s高可用集群部署）
qq_38664479的博客
10-16 3474
目录一、k8s高可用集群介绍1. 实验环境二、K8s高可用+负载均衡集群部署1. haproxy负载均衡部署2. Docker部署3. k8s集群部署4. k8s添加worker节点5. 集群高可用性能测试 一、k8s高可用集群介绍 在前面k8s学习中，围绕一个k8s的master节点操作，当此节点down掉后k8s将无法进行后续的部署管理工作。我们将通过haproxy配置三台 master主机实现负载均衡，通过k8s三台master主机实现k8s集群高可用。 1、Kubernetes的存储层使用的是Et
K8s集群部署之高可用
weixin_33724570的博客
03-23 738
2019独角兽企业重金招聘Python工程师标准>>> ...
FailedScheduling 4m6s (x73 over 109m) default-scheduler 0/12 nodes are available: 12 node(s) didn t match Pod s node affinity
05-29
这个错误信息表示您的 Pod 没有匹配到任何可用的 Node，可能是因为它的 node affinity 配置不正确。Node affinity 是 Kubernetes 中一种配置 Pod 调度规则的方式，可以指定 Pod 只能被调度到特定的 Node 上，或者不能被调度到特定的 Node 上。建议您检查 Pod 的 node affinity 配置是否正确，并且检查集群中是否有可用的 Node。您可以使用 `kubectl get nodes` 命令查看节点的状态。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值