Kubespray 部署 k8s 集群

Kubespray 是 Kubernetes incubator 中的项目，目标是提供 Production Ready Kubernetes 部署方案，该项目基础是通过 Ansible Playbook 来定义系统与 Kubernetes 集群部署的任务，具有以下几个特点：

• 可以部署在 AWS, GCE, Azure, OpenStack 以及裸机上.
• 部署 High Available Kubernetes 集群.
• 可组合性 (Composable)，可自行选择 Network Plugin (flannel, calico, canal, weave) 来部署.
• 支持多种 Linux distributions(CoreOS, Debian Jessie, Ubuntu 16.04, CentOS/RHEL7).

一、环境信息

说明	ip	操作系统
部署机器	172.19.35.97	CentOS 7.7
node1	172.19.35.98	CentOS 7.7
node2	172.19.35.99	CentOS 7.7
node3	172.19.35.100	CentOS 7.7

 

 

二、环境准备(所有主机均安装)

1）所以的主机都需要关闭selinux，执行的命令如下：

setenforce 0
sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

2）防火墙（可选）和网络设置，所有的主机都执行以下命令：

systemctl stop firewalld & systemctl disable firewalld
modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
sysctl -w net.ipv4.ip_forward=1

3）#设置内核参数

sudo vim /etc/security/limits.conf
* soft nofile 32768
* hard nofile 65535
* soft nproc 32768
* hadr nproc 65535

4）设置k8s内核参数

sudo vim /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
vm.swappiness=0

5）重新加载生效

sudo sysctl --system
sudo sysctl -p

6) 安装 python 及 epel 

yum install -y epel-release python36 python36-pip git

三、在Ansible主机上安装并配置好与各node的免秘钥登录

ssh-keygen  #使用 ssh-keygen 命令，一直按回车，就可以生成当前机器的公钥 id_rsa.pub ，
cp .~/ssh/id_rsa.pub  /home/user2/.ssh/id_rsa.pub.user1   # user2 家目录下如果没有 .ssh ，可以使用mkdir 创建

cat ~/.ssh/id_rsa.pub.user1 >> ~/.ssh/authorized_keys  #如果authorized_keys  不存在使用touch 创建
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

四、部署k8s集群

1）#克隆项目

wget https://github.com/kubernetes-sigs/kubespray/archive/v2.12.4.tar.gz

2）# Install dependencies from ``requirements.txt``

sudo /usr/bin/pip3.6 install -r requirements.txt

3）# Copy ``inventory/sample`` as ``inventory/mycluster``

cp -rfp inventory/sample inventory/mycluster

4）# Update Ansible inventory file with inventory builder

declare -a IPS=(172.19.35.98 172.19.35.99 172.19.35.100)
CONFIG_FILE=inventory/mycluster/hosts.yaml /usr/bin/python3.6 contrib/inventory_builder/inventory.py ${IPS[@]}

5）# Review and change parameters under ``inventory/mycluster/group_vars``

cat inventory/mycluster/group_vars/all/all.yml

cat inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml

6）# Deploy Kubespray with Ansible Playbook - run the playbook as root

# The option `--become` is required, as for example writing SSL keys in /etc/,

# installing packages and interacting with various systemd daemons.

# Without --become the playbook will fail to run!

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml

 注意：

1、默认从国外下载镜像因为node需要能上外网。离线部署

2、安装需要点时间耐心等待，遇到报错解决后继续安装。

7) # Reset kubernetes cluster

ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root reset.yml

五、报错解决:

1. 安装 requirements.txt 时候报错 ModuleNotFoundError: No module named 'setuptools_rust'

解决方式: python3  -m pip install --upgrade pip

运维经验

如果需要扩容Work节点，则修改hosts.ini文件，增加新增的机器信息。然后执行下面的命令：

ansible-playbook -i inventory/mycluster/hosts.yaml scale.yml -b -v -k

将hosts.ini文件中的master和etcd的机器增加到多台，执行部署命令

ansible-playbook -i inventory/mycluster/hosts.yaml cluster.yml -b -vvv

刪除节点，如果不指定节点就是刪除整个集群：

ansible-playbook -i inventory/mycluster/hosts.yaml reset.yml -b –vvv

如果需要卸载，可以执行以下命令：

ansible-playbook -i inventory/mycluster/hosts.yaml reset.yml -b –vvv

升级K8s集群，选择对应的k8s版本信息，执行升级命令。涉及文件为upgrade-cluster.yml。

ansible-playbook upgrade-cluster.yml -b -i inventory/mycluster/hosts.yaml -e kube_version=vX.XX.XX -vvv

默认镜像是从国外下载,如果没有翻墙的话,可以走私有网络下载镜像比如push 到docker hub 或者自己搭建的registry 等镜像仓库中,并修改镜像仓库地址

修改配置文件main.yml，使用sed命令批量替换

vim roles/download/defaults/main.yml

修改代码，使用NodePort方式访问Dashboard。

vim ./roles/kubernetes-apps/ansible/templates/dashboard.yml.j2

添加 type: NodePort

注意

如果是单节点部署 K8s，Kubespray默认会创建 2个coredns Pod，但 Deployment中又用到了podAntiAffinity，因此会导致其中一个coredns pod pending，所以需要修改代码如下

vim ./roles/kubernetes-apps/ansible/templates/coredns-deployment.yml.j2

//注释掉以下几行代码

  affinity:
     #  podAntiAffinity:
     #     requiredDuringSchedulingIgnoredDuringExecution:
     #     - topologyKey: "kubernetes.io/hostname"
     #       labelSelector:
     #         matchLabels:
     #           k8s-app: kube-dns{{ coredns_ordinal_suffix }}

或者副本数目设置成1

spec:

  replicas: 1   //指定pod为1个副本

参考文档:

 GitHub地址：https://github.com/kubernetes-sigs/kubespray

Kubespray – 10 Simple Steps for Installing a Production-Ready, Multi-Master HA Kubernetes Cluster:https://dzone.com/articles/kubespray-10-simple-steps-for-installing-a-product

Installing Kubernetes On-premises/Cloud Providers with :https://kubernetes.io/docs/setup/production-environment/tools/kubespray/

https://developer.aliyun.com/article/505382

https://www.jianshu.com/p/d8bee7c8a1e6