kubesphere高可用部署Kubernetes踩的坑

环境介绍

• nfs做后端存储
• haproxy、keepalived做负载均衡
• 10.10.100.201-203 master节点
• 10.10.100.204-207 work节点

环境准备

依赖安装

• nfs-utils NFS客户端
• ntpdate用于时间同步

yum install -y nfs-utils ntpdate

不要照搬kubesphere官网文档中NFS服务器搭建，如果你是centos系统,/etc/exports也添加上no_subtree_check那你估计有得折腾了。也有可能是我脸黑，在创建NFS storageClass的时候提示超级坏快，去掉就好了

• no_subtree_check：此选项阻止子树检查。当共享目录是较大文件系统的子目录时，nfs 会对其上方的每个目录执行扫描，以验证其权限和详细信息。禁用子树检查可能会提高 NFS 的可靠性，但会降低安全性。

kubesphere依赖``

依赖项	Kubernetes 版本 ≥ 1.18	Kubernetes 版本 < 1.18
socat	必须	可选但建议
conntrack	必须	可选但建议
ebtables	可选但建议	可选但建议
ipset	可选但建议	可选但建议

yum install -y socat conntrack ebtables ipset

配置负载均衡

yum install keepalived haproxy psmisc -y

haproxy配置

选择两台work节点复用

vim /etc/haproxy/haproxy.cfg

修改配置文件

global
    log /dev/log  local0 warning
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

   stats socket /var/lib/haproxy/stats

defaults
  log global
  option  httplog
  option  dontlognull
        timeout connect 5000
        timeout client 50000
        timeout server 50000
listen admin_stats
        stats   enable
        bind    *:8080    #监听的ip端口号
        mode    http    #开关
        option  httplog
        log     global
        maxconn 10
        stats   refresh 15s   #统计页面自动刷新时间
        stats   uri /admin    #访问的uri   ip:8080/admin
        stats   realm haproxy
        stats   auth admin:admin  #认证用户名和密码
        stats   hide-version   #隐藏HAProxy的版本号
        stats   admin if TRUE   #管理界面，如果认证成功了，可通过webui管理节点

frontend kube-apiserver
  bind 0.0.0.0:6443
  mode tcp
  option tcplog
  default_backend kube-apiserver

backend kube-apiserver
    mode tcp
    option tcplog
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server kube-apiserver-1 10.10.100.201:6443 check # Replace the IP address with your own.
    server kube-apiserver-2 10.10.100.202:6443 check # Replace the IP address with your own.
    server kube-apiserver-3 10.10.100.203:6443 check # Replace the IP address with your own.

重启并开启自动运行

systemctl restart haproxy
systemctl enable haproxy

keepalived配置

编辑配置文件

vim /etc/keepalived/keepalived.conf

global_defs {
  notification_email {
  }
  router_id LVS_DEVEL
  vrrp_skip_check_adv_addr
  vrrp_garp_interval 0
  vrrp_gna_interval 0
}

vrrp_script chk_haproxy {
  script "killall -0 haproxy"
  interval 2
  weight -20
}

vrrp_instance haproxy-vip {
  state MASTER			# 其它节点为BACKUP
  priority 100
  interface ens192                       # Network card
  virtual_router_id 60
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass 1111
  }
  unicast_src_ip 10.10.100.204      # The IP address of this machine
  unicast_peer {
    10.10.100.205                         # The IP address of peer machines
  }

  virtual_ipaddress {
    10.10.100.251/24                  # The VIP address
  }

  track_script {
    chk_haproxy
  }
}

重启并开启自动运行

systemctl restart keepalived
systemctl enable keepalived

安装containerd

安装依赖

yum install -y yum-utils device-mapper-persistent-data lvm2

添加软件源

yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i 's+download.docker.com+mirrors.aliyun.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo

更新并安装containerd

yum makecache fast
yum install containerd

配置containerd

• 生成默认配置文件

containerd config default > /etc/containerd/config.toml

• 修改配置文件

grep sandbox_image /etc/containerd/config.toml
    sandbox_image = "registry.k8s.io/pause:3.6"	# 默认为registry.k8s.io
    sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"
sed -i "s#registry.k8s.io#/registry.cn-hangzhou.aliyuncs.com/google_containers#g" /etc/containerd/config.toml