基于 haproxy 和 keepalived 部署 kubernetes 高可用集群

willops

已于 2024-01-10 10:07:36 修改

阅读量1.5w

点赞数 10

分类专栏： kubernetes 文章标签： kubernetes

于 2019-04-27 10:59:37 首次发布

本文链接：https://blog.csdn.net/networken/article/details/89599004

版权

kubernetes 专栏收录该内容

50 篇文章 22 订阅

订阅专栏

高可用集群简介

Kubernetes HA 集群搭建，主要包含 Etcd HA 和 Master HA。Etcd HA 通过搭建 Etcd 集群即可（注意 Etcd 集群只能有奇数个节点）。Master HA 多主就是多个 Kubernetes Master 节点组成，任意一个 Master 挂掉后，自动切换到另一个备用 Master，可以采用 haproxy + keepalived 的方案实现负载均衡，然后使用 VIP(虚地址) 方式来实现高可用。

对于从虚拟IP提供负载均衡，keepalived和haproxy的组合已经有很多用例：

keepalived提供了一个由可配置的健康检查管理的虚拟IP。由于虚拟IP的实施方式，协商虚拟IP的所有主机必须在同一个IP子网中。
haproxy服务可以配置为简单的基于流的负载平衡，从而允许TLS终止由其后面的API服务器实例处理。

这种组合既可以作为操作系统上的服务运行，也可以在master节点上作为静态Pods运行。两种情况下的配置是相同的。

基本架构如下图所示：
在这里插入图片描述

环境信息

本示例中的安装部署Kubernetes集群将基于以下环境进行：

OS: Ubuntu Server 22.04 LTS
Kubernetes：v1.28.1
Container Runtime: Containerd

前置要求：

至少2个CPU、2G内存
禁用swap交换分区
允许 iptables 检查桥接流量

示例集群有3个主节点，1个工作节点，3个用于负载均衡的节点与3个k8s 主节点合并部署，以及一个虚拟 IP 地址。本示例中的虚拟 IP 地址也可称为“浮动 IP 地址”。这意味着在节点故障的情况下，该 IP 地址可在节点之间漂移，从而实现高可用。

节点清单：

节点名称	节点IP	节点角色	CPU	内存	磁盘	OS
master1	192.168.72.30	master	2C	4G	100G	Ubutnu 22.04 LTS
master2	192.168.72.31	master	2C	4G	100G	Ubutnu 22.04 LTS
master3	192.168.72.32	master	2C	4G	100G	Ubutnu 22.04 LTS
node1	192.168.72.33	worker	2C	4G	100G	Ubutnu 22.04 LTS
keepalived-vip	192.168.72.200

节点初始化

说明：以下操作在所有节点执行。

1、配置主机名

hostnamectl set-hostname master1
hostnamectl set-hostname master2
hostnamectl set-hostname master3
hostnamectl set-hostname node1

2、配置hosts解析

cat >> /etc/hosts << EOF
192.168.72.30 master1
192.168.72.31 master2
192.168.72.32 master3
192.168.72.33 node1
EOF

3、关闭swap

sed -ri '/\sswap\s/s/^#?/#/' /etc/fstab
mount -a
swapoff -a

4、确认时间同步

apt install -y chrony
systemctl enable --now chrony
chronyc sources

5、加载ipvs内核模块
参考：https://github.com/kubernetes/kubernetes/tree/master/pkg/proxy/ipvs

另外，针对Linux kernel 4.19以上的内核版本使用nf_conntrack 代替nf_conntrack_ipv4。

cat <<EOF | tee /etc/modules-load.d/ipvs.conf
# Load IPVS at boot
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack

#确认内核模块加载成功
lsmod | grep -e ip_vs -e nf_conntrack

#安装ipset和ipvsadm
apt install -y ipset ipvsadm

安装containerd

说明：以下操作在所有节点执行。

1、安装containerd容器运行时的前置条件

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# 设置必需的 sysctl 参数，这些参数在重新启动后仍然存在。
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# 应用 sysctl 参数而无需重新启动
sudo sysctl --system

2、安装containerd容器运行时，如果网络较差，建议使用浏览器下载到本地，在上传到服务器。

下载地址：https://github.com/containerd/nerdctl/releases

wget https://github.com/containerd/nerdctl/releases/download/v1.5.0/nerdctl-full-1.5.0-linux-amd64.tar.gz
tar Cxzvvf /usr/local nerdctl-full-1.5.0-linux-amd64.tar.gz

3、创建containerd配置文件

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

4、配置使用 systemd cgroup 驱动程序

sed -i "s#SystemdCgroup = false#SystemdCgroup = true#g" /etc/containerd/config.toml

5、修改基础设施镜像，安装kubeadm后使用kubeadm config images list命令确认pause默认镜像tag

old_image='sandbox_image = "registry.k8s.io/pause:.*"'
new_image='sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"'
sed -i "s|$old_image|$new_image|" /etc/containerd/config.toml

6、启动containerd服务

systemctl enable --now containerd

7、查看containerd运行状态

systemctl status containerd

安装kubeadm

说明：以下操作在所有节点执行。

1、添加kubernetes源，使用阿里云apt源进行替换：

apt update -y
apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

2、查看可安装的版本

apt-get update
apt-cache madison kubectl | more

3、安装指定版本kubeadm、kubelet及kubectl

export KUBERNETES_VERSION=1.28.1-00
apt update -y
apt-get install -y kubelet=${KUBERNETES_VERSION} kubeadm=${KUBERNETES_VERSION} kubectl=${KUBERNETES_VERSION}
apt-mark hold kubelet kubeadm kubectl

4、启动kubelet服务

systemctl enable --now kubelet

部署haproxy和keepalived

说明：以下操作在所有master节点执行。

1、安装haproxy和keepalived

apt install -y haproxy keepalived

2、创建haproxy配置文件，3个master节点配置相同，注意修改变量适配自身机器环境

export APISERVER_DEST_PORT=6444
export APISERVER_SRC_PORT=6443
export MASTER1_ADDRESS=192.168.72.30
export MASTER2_ADDRESS=192.168.72.31
export MASTER3_ADDRESS=192.168.72.32

cp /etc/haproxy/haproxy.cfg{,.bak}
cat >/etc/haproxy/haproxy.cfg<<EOF
global
  log         127.0.0.1 local0
  log         127.0.0.1 local1 notice
  maxconn     20000
  daemon
  spread-checks 2

defaults
  mode                    http
  log                     global
  option                  tcplog
  option                  dontlognull
  option http-server-close
  option                  redispatch
  timeout http-request    2s
  timeout queue           3s
  timeout connect         1s
  timeout client          1h
  timeout server          1h
  timeout http-keep-alive 1h
  timeout check           2s
  maxconn                 18000 

backend stats-back
  mode http
  balance roundrobin
  stats uri /haproxy/stats
  stats auth admin:1111

frontend stats-front
  bind *:8081
  mode http
  default_backend stats-back

frontend apiserver
  bind *:${APISERVER_DEST_PORT}
  mode tcp
  option tcplog
  default_backend apiserver

backend apiserver
  mode tcp
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server kube-apiserver-1 ${MASTER1_ADDRESS}:${APISERVER_SRC_PORT} check
  server kube-apiserver-2 ${MASTER2_ADDRESS}:${APISERVER_SRC_PORT} check
  server kube-apiserver-3 ${MASTER3_ADDRESS}:${APISERVER_SRC_PORT} check
EOF

3、创建keepalived配置文件，3个master节点配置相同，注意修改变量适配自身机器环境

export APISERVER_VIP=192.168.72.200
export INTERFACE=ens33
export ROUTER_ID=51

cp /etc/keepalived/keepalived.conf{,.bak}
cat >/etc/keepalived/keepalived.conf<<EOF
global_defs {
    router_id ${ROUTER_ID}
    vrrp_version 2
    vrrp_garp_master_delay 1
    vrrp_garp_master_refresh 1
    script_user root
    enable_script_security
}

vrrp_script check_apiserver {
    script "/usr/bin/killall -0 haproxy"
    timeout 3
    interval 5   # check every 5 second
    fall 3       # require 3 failures for KO
    rise 2       # require 2 successes for OK
}

vrrp_instance lb-kube-vip {
    state BACKUP
    interface ${INTERFACE}
    virtual_router_id ${ROUTER_ID}
    priority 51
    advert_int 1
    nopreempt
    track_script {
        check_apiserver
    }
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        ${APISERVER_VIP} dev ${INTERFACE}
    }
}
EOF

说明：这里所有节点都为BACKUP状态，由keepalvied根据优先级自行选举master节点。

4、启动haproxy和keepalived服务

systemctl enable --now haproxy
systemctl enable --now keepalived

5、检查3个master节点，确认VIP地址192.168.72.200生成在哪个节点上

以下示例显示VIP绑定在master1节点的ens33网卡上，可以通过重启该节点确认VIP是否能够自动切换到其他节点。

root@master1:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:aa:75:9f brd ff:ff:ff:ff:ff:ff
    altname enp2s1
    inet 192.168.72.30/24 brd 192.168.72.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.72.200/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feaa:759f/64 scope link 
       valid_lft forever preferred_lft forever

提前拉取k8s镜像

备注：以下操作在所有节点执行。

默认初始化集群时kubeadm能够自动拉取镜像，提前拉取镜像能够缩短集群初次初始化的时间，该操作为可选项。

1、查看可安装的kubernetes版本

kubectl version --short

2、查看对应kubernetes版本的容器镜像

由于registry.k8s.io 项目由谷歌 GCP 和 AWS 托管并捐赠支持，registry.k8s.io已被屏蔽，K8S官方社区表示无能为力，因此需要通过--image-repository参数指定使用国内阿里云k8s镜像仓库。

kubeadm config images list \
  --kubernetes-version=v1.28.1 \
  --image-repository registry.aliyuncs.com/google_containers

3、在所有节点执行以下命令，提前拉取镜像

kubeadm config images pull \
  --kubernetes-version=v1.28.1 \
  --image-repository registry.aliyuncs.com/google_containers

4、查看拉取的镜像

root@master1:~# nerdctl -n k8s.io images |grep -v none
REPOSITORY                                                         TAG        IMAGE ID        CREATED               PLATFORM       SIZE         BLOB SIZE
registry.aliyuncs.com/google_containers/coredns                    v1.10.1    90d3eeb2e210    About a minute ago    linux/amd64    51.1 MiB     15.4 MiB
registry.aliyuncs.com/google_containers/etcd                       3.5.9-0    b124583790d2    About a minute ago    linux/amd64    283.8 MiB    98.1 MiB
registry.aliyuncs.com/google_containers/kube-apiserver             v1.28.1    1e9a3ea7d1d4    2 minutes ago         linux/amd64    123.1 MiB    33.0 MiB
registry.aliyuncs.com/google_containers/kube-controller-manager    v1.28.1    f6838231cb74    2 minutes ago         linux/amd64    119.5 MiB    31.8 MiB
registry.aliyuncs.com/google_containers/kube-proxy                 v1.28.1    feb6017bf009    About a minute ago    linux/amd64    73.6 MiB     23.4 MiB
registry.aliyuncs.com/google_containers/kube-scheduler             v1.28.1    b76ea016d6b9    2 minutes ago         linux/amd64    60.6 MiB     17.9 MiB
registry.aliyuncs.com/google_containers/pause                      3.9        7031c1b28338    About a minute ago    linux/amd64    728.0 KiB    314.0 KiB

创建集群配置文件

说明：以下操作仅在第一个master节点执行。

1、生成默认的集群初始化配置文件。

kubeadm config print init-defaults --component-configs KubeProxyConfiguration > kubeadm.yaml

2、修改集群配置文件

$ cat kubeadm.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.72.30
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: master1
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
  certSANs:
  - master1
  - master2
  - master3
  - node1
  - 192.168.72.200
  - 192.168.72.30
  - 192.168.72.31
  - 192.168.72.32
  - 192.168.72.33
controlPlaneEndpoint: 192.168.72.200:6444
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.28.1
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16
scheduler: {}
---
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
---
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
bindAddressHardFail: false
clientConnection:
  acceptContentTypes: ""
  burst: 0
  contentType: ""
  kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
  qps: 0
clusterCIDR: ""
configSyncPeriod: 0s
conntrack:
  maxPerCore: null
  min: null
  tcpCloseWaitTimeout: null
  tcpEstablishedTimeout: null
detectLocal:
  bridgeInterface: ""
  interfaceNamePrefix: ""
detectLocalMode: ""
enableProfiling: false
healthzBindAddress: ""
hostnameOverride: ""
iptables:
  localhostNodePorts: null
  masqueradeAll: false
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s
ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: ""
  strictARP: false
  syncPeriod: 0s
  tcpFinTimeout: 0s
  tcpTimeout: 0s
  udpTimeout: 0s
kind: KubeProxyConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
  verbosity: 0
metricsBindAddress: ""
mode: "ipvs"
nodePortAddresses: null
oomScoreAdj: null
portRange: ""
showHiddenMetricsForVersion: ""
winkernel:
  enableDSR: false
  forwardHealthCheckVip: false
  networkName: ""
  rootHnsEndpointName: ""
  sourceVip: ""

3、在默认值基础之上需要配置的参数说明：

InitConfiguration

kind: InitConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
localAPIEndpoint:
  advertiseAddress: 192.168.72.30
  bindPort: 6443
nodeRegistration:

  name: master1

ClusterConfiguration

kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
controlPlaneEndpoint: 192.168.72.200:6444
imageRepository: registry.aliyuncs.com/google_containers
kubernetesVersion: 1.28.1
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16
apiServer:
  certSANs:
  - master1
  - master2
  - master3
  - node1
  - 192.168.72.200
  - 192.168.72.30
  - 192.168.72.31
  - 192.168.72.32
  - 192.168.72.33

KubeletConfiguration

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""

KubeProxyConfiguration

kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
mode: "ipvs"

初始化第一个master节点

1、在第一个master节点运行以下命令开始初始化master节点：

kubeadm init --upload-certs --config kubeadm.yaml

如果初始化报错可以执行以下命令检查kubelet相关日志。

journalctl -xeu kubelet

2、记录日志输出中的join control-plane和join worker命令。

......
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.72.200:6444 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:293b145b86ee0839a650befd4d32d706852ac5c77848a55e0cb186be29ff38de \
        --control-plane --certificate-key 578ad0ee6a1052703962b0a8591d0036f23a514d4456bd08fa253eda00128fca

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.72.200:6444 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:293b145b86ee0839a650befd4d32d706852ac5c77848a55e0cb186be29ff38de

3、master节点初始化完成后参考最后提示配置kubectl客户端连接

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

4、查看节点状态，当前还未安装网络插件节点处于NotReady状态

root@master1:~# kubectl get nodes
NAME   STATUS     ROLES           AGE   VERSION
node   NotReady   control-plane   40s   v1.28.1

5、查看pod状态，当前还未安装网络插件coredns pod处于Pending状态

root@master1:~# kubectl get pods -A
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
kube-system   coredns-66f779496c-44fcc       0/1     Pending   0          32s
kube-system   coredns-66f779496c-9cjmf       0/1     Pending   0          32s
kube-system   etcd-node                      1/1     Running   1          44s
kube-system   kube-apiserver-node            1/1     Running   1          44s
kube-system   kube-controller-manager-node   1/1     Running   1          44s
kube-system   kube-proxy-g4kns               1/1     Running   0          32s
kube-system   kube-scheduler-node            1/1     Running   1          44s

安装calico网络插件

说明：以下操作仅在第一个master节点执行。

参考：https://projectcalico.docs.tigera.io/getting-started/kubernetes/quickstart

1、在第一个master节点安装helm

version=v3.12.3
curl -LO https://repo.huaweicloud.com/helm/${version}/helm-${version}-linux-amd64.tar.gz
tar -zxvf helm-${version}-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm && rm -rf linux-amd64

2、添加calico helm 仓库

helm repo add projectcalico https://projectcalico.docs.tigera.io/charts

3、部署calico，如果无法访问dockerhub，可能需要提前拉取镜像

helm install calico projectcalico/tigera-operator \
--namespace tigera-operator --create-namespace

4、查看节点变为Ready状态

root@master1:~# kubectl get nodes
NAME   STATUS   ROLES           AGE   VERSION
node   Ready    control-plane   17m   v1.28.1

5、查看coredns pod状态变为Running

root@master1:~# kubectl get pods -A
NAMESPACE          NAME                                      READY   STATUS    RESTARTS   AGE
calico-apiserver   calico-apiserver-c7b9c94d5-ck6n8          1/1     Running   0          4m6s
calico-apiserver   calico-apiserver-c7b9c94d5-fjbbh          1/1     Running   0          4m6s
calico-system      calico-kube-controllers-9c4d4576f-rtbm5   1/1     Running   0          11m
calico-system      calico-node-vwkhs                         1/1     Running   0          11m
calico-system      calico-typha-76c649f99d-7gdsv             1/1     Running   0          11m
calico-system      csi-node-driver-fzlck                     2/2     Running   0          11m
kube-system        coredns-66f779496c-44fcc                  1/1     Running   0          17m
kube-system        coredns-66f779496c-9cjmf                  1/1     Running   0          17m
kube-system        etcd-node                                 1/1     Running   1          17m
kube-system        kube-apiserver-node                       1/1     Running   1          17m
kube-system        kube-controller-manager-node              1/1     Running   1          17m
kube-system        kube-proxy-g4kns                          1/1     Running   0          17m
kube-system        kube-scheduler-node                       1/1     Running   1          17m
tigera-operator    tigera-operator-94d7f7696-lzrrz           1/1     Running   0          11m

master节点加入集群

1、分别在第2和第3个master节点执行以下命令

kubeadm join 192.168.72.200:6444 --token abcdef.0123456789abcdef \
      --discovery-token-ca-cert-hash sha256:7e25d6bc9368f5ab747c8c0b3349c5441005e0e15914cd4be1eac6351fb1e320 \
      --control-plane --certificate-key fbae5ace48cf806fe5df818174859e3a837cf316156cd6e26ec14c7fca28fba3

2、如果master初始化后未记录节点加入集群命令，可以通过运行以下命令重新生成：

kubeadm token create --print-join-command --ttl 0

worker节点加入集群

1、在node1节点上执行如下命令，将其注册到 Cluster 中：

kubeadm join 192.168.72.200:6444 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:7e25d6bc9368f5ab747c8c0b3349c5441005e0e15914cd4be1eac6351fb1e320

2、通过 kubectl get nodes 查看节点的状态。

root@master1:~# kubectl get nodes -o wide
NAME      STATUS   ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master1   Ready    control-plane   22m     v1.28.1   192.168.72.30   <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd://1.7.3
master2   Ready    control-plane   4m34s   v1.28.1   192.168.72.31   <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd://1.7.3
master3   Ready    control-plane   3m48s   v1.28.1   192.168.72.32   <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd://1.7.3
node1     Ready    <none>          3m41s   v1.28.1   192.168.72.33   <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd://1.7.3

3、最终运行的pods

root@master1:~# kubectl get pods -A -o wide
NAMESPACE          NAME                                       READY   STATUS    RESTARTS      AGE   IP               NODE      NOMINATED NODE   READINESS GATES
calico-apiserver   calico-apiserver-866fdb498d-2jndc          1/1     Running   0             18m   10.244.137.69    master1   <none>           <none>
calico-apiserver   calico-apiserver-866fdb498d-672gj          1/1     Running   0             18m   10.244.137.70    master1   <none>           <none>
calico-system      calico-kube-controllers-57b4847fc8-zlf8t   1/1     Running   0             18m   10.244.137.66    master1   <none>           <none>
calico-system      calico-node-bxjhp                          1/1     Running   0             18m   192.168.72.30    master1   <none>           <none>
calico-system      calico-node-j2zxw                          1/1     Running   0             12m   192.168.72.31    master2   <none>           <none>
calico-system      calico-node-kjfhh                          1/1     Running   0             11m   192.168.72.33    node1     <none>           <none>
calico-system      calico-node-zmngb                          1/1     Running   0             11m   192.168.72.32    master3   <none>           <none>
calico-system      calico-typha-548dd44bd-hzj8b               1/1     Running   0             18m   192.168.72.30    master1   <none>           <none>
calico-system      calico-typha-548dd44bd-pjpn5               1/1     Running   0             11m   192.168.72.32    master3   <none>           <none>
calico-system      csi-node-driver-48xkx                      2/2     Running   0             18m   10.244.137.65    master1   <none>           <none>
calico-system      csi-node-driver-rtxt2                      2/2     Running   0             11m   10.244.136.1     master3   <none>           <none>
calico-system      csi-node-driver-ssfbh                      2/2     Running   0             11m   10.244.166.129   node1     <none>           <none>
calico-system      csi-node-driver-v8h9l                      2/2     Running   0             12m   10.244.180.1     master2   <none>           <none>
kube-system        coredns-66f779496c-ldfpv                   1/1     Running   0             23m   10.244.137.68    master1   <none>           <none>
kube-system        coredns-66f779496c-zx8rs                   1/1     Running   0             23m   10.244.137.67    master1   <none>           <none>
kube-system        etcd-master1                               1/1     Running   0             23m   192.168.72.30    master1   <none>           <none>
kube-system        etcd-master2                               1/1     Running   0             12m   192.168.72.31    master2   <none>           <none>
kube-system        etcd-master3                               1/1     Running   0             12m   192.168.72.32    master3   <none>           <none>
kube-system        kube-apiserver-master1                     1/1     Running   0             23m   192.168.72.30    master1   <none>           <none>
kube-system        kube-apiserver-master2                     1/1     Running   0             12m   192.168.72.31    master2   <none>           <none>
kube-system        kube-apiserver-master3                     1/1     Running   1 (12m ago)   12m   192.168.72.32    master3   <none>           <none>
kube-system        kube-controller-manager-master1            1/1     Running   1 (12m ago)   23m   192.168.72.30    master1   <none>           <none>
kube-system        kube-controller-manager-master2            1/1     Running   0             12m   192.168.72.31    master2   <none>           <none>
kube-system        kube-controller-manager-master3            1/1     Running   0             11m   192.168.72.32    master3   <none>           <none>
kube-system        kube-proxy-4dzjw                           1/1     Running   0             23m   192.168.72.30    master1   <none>           <none>
kube-system        kube-proxy-cqkbr                           1/1     Running   0             11m   192.168.72.32    master3   <none>           <none>
kube-system        kube-proxy-vcr8w                           1/1     Running   0             11m   192.168.72.33    node1     <none>           <none>
kube-system        kube-proxy-x6tkn                           1/1     Running   0             12m   192.168.72.31    master2   <none>           <none>
kube-system        kube-scheduler-master1                     1/1     Running   1 (12m ago)   23m   192.168.72.30    master1   <none>           <none>
kube-system        kube-scheduler-master2                     1/1     Running   0             12m   192.168.72.31    master2   <none>           <none>
kube-system        kube-scheduler-master3                     1/1     Running   0             12m   192.168.72.32    master3   <none>           <none>
tigera-operator    tigera-operator-94d7f7696-b5fbl            1/1     Running   1 (12m ago)   19m   192.168.72.30    master1   <none>           <none>

验证集群高可用

1、关闭master1节点

root@master1:~# shutdown -h now

2、查看vip地址自动转移到master3节点

root@master3:~# ip a |grep ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 192.168.72.32/24 brd 192.168.72.255 scope global ens33
    inet 192.168.72.200/32 scope global ens33

3、在master3节点配置kubectl客户端连接信息

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

4、在master3节点验证依然能够正常访问集群

root@master3:~# kubectl get nodes
NAME      STATUS     ROLES           AGE   VERSION
master1   NotReady   control-plane   39m   v1.28.1
master2   Ready      control-plane   27m   v1.28.1
master3   Ready      control-plane   27m   v1.28.1
node1     Ready      <none>          27m   v1.28.1

至此，基于haproxy和keepalived的高可用集群部署完成。

willops

关注

10
点赞
踩
54

收藏

觉得还不错? 一键收藏
10
评论
基于 haproxy 和 keepalived 部署 kubernetes 高可用集群

Kubernetes HA 集群搭建，主要包含 Etcd HA 和 Master HA。Etcd HA 通过搭建 Etcd 集群即可（注意 Etcd 集群只能有奇数个节点）。Master HA 多主就是多个 Kubernetes Master 节点组成，任意一个 Master 挂掉后，自动切换到另一个备用 Master，可以采用 haproxy + keepalived 的方案实现负载均衡，然后使用 VIP(虚地址) 方式来实现高可用。keepalived提供了一个由可配置的健康检查管理的虚拟IP。
复制链接

扫一扫