kubeasz部署高可用kubernetes集群

准备

工具

https://github.com/easzlab/kubeasz

kubeasz 使用 ansible 快速部署非容器化 高可用 k8s 集群

kubeasz 主要有 1.x 和 2.x 两个版本，版本比较参考：https://github.com/easzlab/kubeasz/blob/master/docs/mixes/branch.md

这里使用 2.x 版本

环境

主机	内网ip	外网ip	系统
k8s-1	10.0.0.18	61.184.241.187	ubuntu 18.04
k8s-2	10.0.0.19		ubuntu 18.04
k8s-3	10.0.0.20		ubuntu 18.04

架构

k8s集群内部访问 apiserver，可以通过节点自身安装的 haproxy 实现负载，高可用，而外部访问 apiserver，是通过另外的 haproxy + keepalived 实现的，这个外部的负载均衡是可选的，因为该负载只影响能不能高可用访问 k8s 管理界面，并不影响 k8s 内部集群的高可用，所以可以选择不安装

参考：https://github.com/easzlab/kubeasz/issues/585#issuecomment-502948966

master与node节点共用时，无法在master上部署haproxy+keepalived实现外部访问apiserver高可用，详情：https://github.com/easzlab/kubeasz/issues/585#issuecomment-575954720

规划

部署节点	k8s-1
etcd 节点	k8s-1	k8s-2	k8s-3
master 节点	k8s-1	k8s-2
node 节点	k8s-1	k8s-2	k8s-3

部署

默认全部使用 root 用户操作

配置 DNS

# 所有节点
vim /etc/hosts
10.0.0.18 k8s-1
10.0.0.19 k8s-2
10.0.0.20 k8s-3

修改 apt 源

# 所有节点
cp /etc/apt/sources.list /etc/apt/sources.list.bakcup
cat > /etc/apt/sources.list <<EOF
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
EOF

apt update

配置ssh免密登陆

# 在部署节点 k8s-1 上
ssh-keygen
ssh-copy-id k8s-1
ssh-copy-id k8s-2
ssh-copy-id k8s-3

内核升级

ubuntu 18.04 使用内核 4.15, 达到要求, 不需要更新, 其他系统内核更新参考:
https://github.com/easzlab/kubeasz/blob/master/docs/guide/kernel_upgrade.md

安装依赖

# 在所有节点上
apt install -y python2.7

下载

cd /opt
# 下载工具脚本easzup，举例使用kubeasz版本2.2.0
export release=2.2.0
curl -C- -fLO --retry 3 https://github.com/easzlab/kubeasz/releases/download/${release}/easzup
chmod +x ./easzup
# 使用工具脚本下载
./easzup -D

上述脚本运行成功后，所有文件（kubeasz代码、二进制、离线镜像）均已整理好放入目录/etc/ansible

• /etc/ansible 包含 kubeasz 版本为 ${release} 的发布代码
• /etc/ansible/bin 包含 k8s/etcd/docker/cni 等二进制文件
• /etc/ansible/down 包含集群安装时需要的离线容器镜像
• /etc/ansible/down/packages 包含集群安装时需要的系统基础软件

安装集群

# 容器化运行 kubeasz
cd /opt
./easzup -S
# 安装集群
# 创建集群上下文
docker exec -it kubeasz easzctl checkout myk8s

# 修改hosts文件
cd /etc/ansible && cp example/hosts.multi-node hosts

# 'etcd' cluster should have odd member(s) (1,3,5,...)
# variable 'NODE_NAME' is the distinct name of a member in 'etcd' cluster
[etcd]
10.0.0.18 NODE_NAME=etcd1
10.0.0.19 NODE_NAME=etcd2
10.0.0.20 NODE_NAME=etcd3

# master node(s)
[kube-master]
10.0.0.18
10.0.0.19

# work node(s)
[kube-node]
10.0.0.18
10.0.0.19
10.0.0.20

# [optional] harbor server, a private docker registry
# 'NEW_INSTALL': 'yes' to install a harbor server; 'no' to integrate with existed one
# 'SELF_SIGNED_CERT': 'no' you need put files of certificates named harbor.pem and harbor-key.pem in directory 'down'
[harbor]
#192.168.1.8 HARBOR_DOMAIN="harbor.yourdomain.com" NEW_INSTALL=no SELF_SIGNED_CERT=yes

# [optional] loadbalance for accessing k8s from outside
[ex-lb]
#192.168.1.6 LB_ROLE=backup EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443
#192.168.1.7 LB_ROLE=master EX_APISERVER_VIP=192.168.1.250 EX_APISERVER_PORT=8443

# [optional] ntp server for the cluster
[chrony]
10.0.0.18

[all:vars]
# --------- Main Variables ---------------
# Cluster container-runtime supported: docker, containerd
CONTAINER_RUNTIME="docker"

# Network plugins supported: calico, flannel, kube-router, cilium, kube-ovn
CLUSTER_NETWORK="flannel"

# Service proxy mode of kube-proxy: 'iptables' or 'ipvs'
PROXY_MODE="ipvs"

# K8S Service CIDR, not overlap with node(host) networking
SERVICE_CIDR="10.68.0.0/16"

# Cluster CIDR (Pod CIDR), not overlap with node(host) networking
CLUSTER_CIDR="172.20.0.0/16"

# NodePort Range
NODE_PORT_RANGE="20000-40000"

# Cluster DNS Domain
CLUSTER_DNS_DOMAIN="cluster.local."

# -------- Additional Variables (don't change the default value right now) ---
# Binaries Directory
bin_dir="/opt/kube/bin"

# CA and other components cert/key Directory
ca_dir="/etc/kubernetes/ssl"

# Deploy Directory (kubeasz workspace)
base_dir="/etc/ansible"

# 配置整个 k8s 集群
docker exec -it kubeasz easzctl setup

验证

# 如果提示kubectl: command not found，退出重新ssh登录一下，环境变量生效即可

$ kubectl version                   # 验证集群版本     
$ kubectl get componentstatus       # 验证 scheduler/controller-manager/etcd等组件状态
$ kubectl get node                  # 验证节点就绪 (Ready) 状态
$ kubectl get pod --all-namespaces  # 验证集群pod状态，默认已安装网络插件、coredns、metrics-server等
$ kubectl get svc --all-namespaces  # 验证集群服务状态

清理

# 清理集群
docker exec -it kubeasz easzctl destroy

# 进一步清理, 清理部署节点上的 docker 以及下载的所有文件
# 清理运行的容器 easzup -C
# 清理容器镜像 docker system prune -a
# 停止docker服务 systemctl stop docker
# 删除下载文件 rm -rf /etc/ansible /etc/docker /opt/kube
# 删除docker文件
$ umount /var/run/docker/netns/default
$ umount /var/lib/docker/overlay
$ rm -rf /var/lib/docker /var/run/docker
# 重启节点