ansible + kubeasz 二进制部署K8S高可用集群方案

K8S架构图

基础环境配置

类型	服务器IP地址	备注
Ansible(2台)	192.168.15.101/102	K8S集群部署服务器，可以和Master节点在共用
K8s Master(2台)	192.168.15.101/102	K8S控制端，通过一个VIP做主备高可用
Harbor(2台)	192.168.15.103/104	高可用镜像服务器
Etcd(最少3台)	192.168.15.105/106/107	保存K8S集群数据的服务器
Haproxy(2台)	192.168.15.108/109	高可用etcd代理服务器
K8s Noder(2-N台)	192.168.15.110/111/…	真正运行容器的服务器，高可用环境至少两台

主机名设置

主机名	服务器IP	VIP
k8s-master01	192.168.15.101	192.168.15.210
k8s-master02	192.168.15.102	192.168.15.210
k8s-harbor01	192.168.15.103
k8s-etcd01	192.168.15.104
k8s-etcd02	192.168.15.105
k8s-etcd03	192.168.15.106
k8s-ha01	192.168.15.107
k8s-ha02	192.168.15.108
k8s-node01	192.168.15.109
k8s-node02	192.168.15.110

服务器基础环境配置参考之前的K8S集群部署博文

harbor镜像仓库之Https

安装docker

# 卸载之前安装过得docker(若之前没有安装过docker,直接跳过此步)
sudo yum remove docker docker-common docker-selinux docker-engine

# 安装docker需要的依赖包 (之前执行过,可以省略)
wget -O /etc/yum.repos.d/docker-ce.repo https://repo.huaweicloud.com/docker-ce/linux/centos/docker-ce.repo

# 安装docker软件
yum install docker-ce -y

# 配置镜像下载加速器
mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
   
  "registry-mirrors": ["https://hahexyip.mirror.aliyuncs.com"]
}
EOF

# 启动docker并加入开机自启动
systemctl enable docker && systemctl start docker

# 查看docker是否成功安装
docker version

在harbor主机安装docker-compose

# 下载安装Docker Compose
curl -L https://download.fastgit.org/docker/compose/releases/download/1.27.4/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose

# 添加执行权限
chmod +x /usr/local/bin/docker-compose 

# 检查安装版本
docker-compose --version 

# bash命令补全
curl -L https://raw.githubusercontent.com/docker/compose/1.25.5/contrib/completion/bash/er-compose > /etc/bash_completion.d/docker-compose

创建CA证书

# 存放密钥的仓库
mkdir -p /usr/local/src/harbor/certs

# 生成私钥key
openssl genrsa -out /usr/local/src/harbor/certs/harbor-ca.key

ls  /usr/local/src/harbor/certs/
# harbor-ca.key

# 签证(生成公钥key)
openssl req -x509 -new -nodes -key /usr/local/src/harbor/certs/harbor-ca.key -subj "/CN=harbor.nana.com" -days 3560 -out /usr/local/src/harbor/certs/harbor-ca.crt

ls  /usr/local/src/harbor/certs/
# harbor-ca.crt  harbor-ca.key

在harbor主机上部署harbor服务

链接：harbor-offline-installer-v2.1.0.tgz
提取码：1234

mkdir /apps
cd /apps
rz -E harbor-offline-installer-v2.1.0.tgz	
tar -xvf harbor-offline-installer-v2.1.0.tgz 
cd harbor/
cp harbor.yml.tmpl harbor.yml

# 修改配置harbor文件
vim harbor.yml 
...
hostname: harbor.nana.com		# harbor的web页面域名
...
  certificate: /usr/local/src/harbor/certs/harbor-ca.crt		# 公钥路径	
  private_key: /usr/local/src/harbor/certs/harbor-ca.key		# 私钥路径
...
harbor_admin_password: 123			# harbor的web页面登陆密码
...
data_volume: /data		# 生产环境最好修改成数据盘

# 安装harbor仓库
./install.sh

client同步crt证书

• k8s-master01主机先安装好docker

# k8s-master01,客户端主机创建存放公钥的目录; 添加域名解析
k8s-master01: mkdir -p /etc/docker/certs.d/harbor.nana.com
k8s-master01: echo 192.168.15.103 harbor.nana.com >> /etc/hosts

# 将harbor主机的公钥发送给k8s-master01主机
k8s-harbor01: scp /usr/local/src/harbor/certs/harbor-ca.crt 192.168.15.101:/etc/docker/certs.d/harbor.nana.com

# docker服务的请求添加域名解析
k8s-master01:
vim /usr/lib/systemd/system/docker.service 
...
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry harbor.nana.com
...

# 重启docker服务
k8s-master01:
systemctl daemon-reload
systemctl restart docker

测试登陆harbor，在k8s-master01主机进行操作

# 登陆到harbor仓库
docker login harbor.nana.com
# 拉取镜像
docker pull alpine
# 给alpine镜像打上标签
docker tag alpine harbor.nana.com/library/alpine:latest
# 上传镜像到harbor仓库
docker push harbor.nana.com/library/alpine:latest

Haproxy+keepalived 高可用负载均衡

配置keepalived

• k8s-ha01主机进行操作

# 下载haproxy keepalived
yum -y install haproxy keepalived

# 配置keepalived
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   
   notification_email {
   
     acassen
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
   
    state MASTER
    interface eth0
    virtual_router_id 55
    priority 100
    advert_int 1
    authentication {
   
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
   
        192.168.15.188 dev eth0 label eth0:1
    }
}

# 启动keepalived服务
systemctl restart keepalived && systemctl enable keepalived

• k8s-ha02主机进行操作

# 下载haproxy keepalived
yum -y install haproxy keepalived

# 配置keepalived
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   
   notification_email {
   
     acassen
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_time