kubespray部署高可用kubernetes集群(国内镜像)

准备

工具

https://github.com/kubernetes-sigs/kubespray

kubespray 使用 ansible 快速部署容器化 高可用 k8s 集群

环境

主机	内网ip	外网ip	系统
k8s-1	10.0.0.18	61.xxx.xxx.187	ubuntu 18.04
k8s-2	10.0.0.19		ubuntu 18.04
k8s-3	10.0.0.20		ubuntu 18.04

规划

部署节点	k8s-1
etcd 节点	k8s-1	k8s-2	k8s-3
master 节点	k8s-1	k8s-2
node 节点	k8s-1	k8s-2	k8s-3

部署

默认全部使用 root 用户操作

配置 DNS

# 所有节点
vim /etc/hosts
10.0.0.18 k8s-1
10.0.0.19 k8s-2
10.0.0.20 k8s-3

修改 apt 源

# 所有节点
cp /etc/apt/sources.list /etc/apt/sources.list.bakcup
cat > /etc/apt/sources.list <<EOF
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
EOF

apt update

配置ssh免密登陆

# 在部署节点 k8s-1 上
ssh-keygen
ssh-copy-id k8s-1
ssh-copy-id k8s-2
ssh-copy-id k8s-3

内核升级

ubuntu 18.04 使用内核 4.15, 达到要求, 不需要更新, 其他系统内核更新参考:
https://github.com/easzlab/kubeasz/blob/master/docs/guide/kernel_upgrade.md

安装依赖

# 在所有节点上
apt install -y python2.7
# 在部署节点 k8s-1 上
apt install -y python3-pip
pip3 install pip --upgrade -i https://mirrors.aliyun.com/pypi/simple/

修改 pip 源

mkdir ~/.pip
cat > ~/.pip/pip.conf << EOF 
[global]
trusted-host=mirrors.aliyun.com
index-url=https://mirrors.aliyun.com/pypi/simple/
EOF

修改 docker 源

# 所有节点上
# 阿里云镜像加速, 可以使用自己的加速器地址
mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://jzngeu7d.mirror.aliyuncs.com"]
}
EOF
systemctl restart docker

kubespray

cd /opt
git clone https://github.com/kubernetes-sigs/kubespray
# 安装依赖
cd kubespray
pip3 install -U setuptools cryptography
pip3 install -r requirements.txt
# 拷贝配置文件
cp -rfp inventory/sample inventory/mycluster
# 使用脚本配置 ansible inventory 文件
declare -a IPS=(10.0.0.18 10.0.0.19 10.0.0.20)
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
# DEBUG: Adding group all
# DEBUG: Adding group kube-master
# DEBUG: Adding group kube-node
# DEBUG: Adding group etcd
# DEBUG: Adding group k8s-cluster
# DEBUG: Adding group calico-rr
# DEBUG: adding host node1 to group all
# DEBUG: adding host node2 to group all
# DEBUG: adding host node3 to group all
# DEBUG: adding host node1 to group etcd
# DEBUG: adding host node2 to group etcd
# DEBUG: adding host node3 to group etcd
# DEBUG: adding host node1 to group kube-master
# DEBUG: adding host node2 to group kube-master
# DEBUG: adding host node1 to group kube-node
# DEBUG: adding host node2 to group kube-node
# DEBUG: adding host node3 to group kube-node

# 修改剧本默认配置
# 主要是 group_vars/all/all.yml 和 group_vars/k8s-cluster/k8s-cluster.yml
# 优先级 k8s-cluster.yml > all.yml > roles/xxx/defalut/main.yml
# 所以想要覆盖 role 里面的默认配置, 优先看 k8s-cluster.yml 里面是否有同名配置, 如果有就同时修改 k8s-cluster.yml 和 all.yml, 没有就在 all.yml 里面添加
# 或者直接使用 ansible-playbook -e @foo.yml 的方式, 因为 -e 指定的变量具有最高优先级
# kubespray 常用变量参考: https://kubespray.io/#/docs/vars?id=common-vars-that-are-used-in-kubespray
vim inventory/mycluster/group_vars/all/all.yml
# 加载内核模块，否则 ceph, gfs 等无法挂载客户端
kubelet_load_modules: true
gcr_image_repo: "gcr.azk8s.cn"
kube_image_repo: "gcr.azk8s.cn/google-containers"
quay_image_repo: "quay.mirrors.ustc.edu.cn"
docker_ubuntu_repo_base_url: "http://mirrors.aliyun.com/docker-ce/linux/ubuntu"
docker_ubuntu_repo_gpgkey: 'http://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg'

vim inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
kube_image_repo: "gcr.azk8s.cn/google-containers"

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml

# 剧本可使用的 tag 参考: https://kubespray.io/#/docs/ansible?id=ansible-tags

# 如果有些github资源下载过慢, 如 https://github.com/containernetworking/plugins/releases/download/v0.8.3/cni-plugins-linux-amd64-v0.8.3.tgz 下载过慢
# 可以配置在 all.yml 中 download_force_cache: true
# 并手动下载好之后放到各节点的 /tmp/kubespray_cache/ 目录, 没有该目录自行创建

管理

增加节点

1. 修改 hosts.yml 文件
2. ansible-playbook -i inventory/mycluster/hosts.yml scale.yml -b

删除节点

1. ansible-playbook -i inventory/mycluster/hosts.yml remove-node.yml -b --extra-vars "node=nodename,nodename2"
2. 如果待删除的节点无法通过 ssh连接，需要添加 --extra-vars reset_nodes=no

组件升级

参考：https://kubespray.io/#/docs/upgrades

HA

https://kubespray.io/#/docs/ha-mode

生产环境上推荐使用外部自建负载均衡，内部负载通过 node 节点上的 nginx 或是 haproxy 实现，与 kubeasz 2.x 架构思想一样

大型部署

https://kubespray.io/#/docs/large-deployments

备注

1. 云上环境尽量使用 flannel网络插件
2. 如果使用calico 或 kube-router 网络插件, 并且在 openstack 上, 需要对集群所有主机的端口设置可用地址对，允许10.233.0.0/18 和 10.233.64.0/18，详情：https://kubespray.io/#/docs/openstack

参考

https://kubespray.io/#/

烂泥：docker.io、gcr.io、quay.io镜像加速