【DevOps】DevOps 实践之路（三）：使用 Ansible 批量安装与配置 K8s

前言

---

在上一篇对所有机器进行了 Docker 的安装配置后，这篇文章我们来批量安装与配置 K8s；

环境

ubuntu 16.04 、ubuntu18.04

docker 19.03.13

Kubernetes v1.19.3

前置条件

在安装 k8s 之前，我们首先要保证每台机器上都已经正确的安装与配置 Docker，可以用如下 ansible 命令查看：

ansible servers_all -m command -a "docker version"
ansible servers_all -m command -a "systemctl status docker -l"
ansible servers_all -m command -a "docker info"

Ansible 批量设置k8s环境的前置条件

---

首先回顾一下手动配置前置条件的步骤，再改写为 playbook 去批量操作；

安装K8s前的准备工作（命令行方式）

1、关闭防火墙

ufw disable
# or
systemctl stop ufw

2、关闭selinux

sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

3、禁用交换分区

sudo swapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab

4、修改 hostname

hostnamectl
hostnamectl set-hostname k8s-174

sudo vim /etc/hosts
xxx.xxx.17.174 k8s-174

改写为playbook

1、批量修改 hostname

sudo vim update_hostname.yml

- hosts: servers_all
  become: true
  gather_facts: False
  tasks:
    - name: get new hostname
      shell: echo k8s-{{ansible_ssh_host.split('.')[-1]}}
      register: hostname
    - name: echo new hostname
      debug:
        msg: "{{hostname.stdout}}"   # 打印变量
    - name: hostnamectl set-hostname
      shell: hostnamectl set-hostname {{hostname.stdout}}
    - name: update /etc/hosts
      shell: echo "{{ansible_ssh_host}} {{hostname.stdout}}" >> /etc/hosts

ansible-playbook update_hostname.yml

ansible servers_all -m shell -a "hostname"

2、批量 关闭防火墙 、关闭 selinux 、禁用交换分区

sudo vim before_install_k8s.yml

- hosts: servers_all
  become: true

  tasks:
    - name: stop ufw
      shell: ufw disable
      ignore_errors: yes

    - name: stop ufw2
      shell: systemctl stop ufw
      ignore_errors: yes

    - name: stop selinux
      shell: apt install selinux-utils && setenforce 0
      ignore_errors: yes

    - name: stop selinux2
      shell: sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux && sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
      ignore_errors: yes

    - name: stop swap
      shell: swapoff -a &&  sed -i 's/.*swap.*/#&/' /etc/fstab

ansible-playbook before_install_k8s.yml

验证一下是否设置成功

ansible servers_all -m shell -a "ufw status"
ansible servers_all -m shell -a "getenforce"
ansible servers_all -m shell -a "free -h"

Ansible 批量安装k8s

---

手动安装的步骤为：

sudo apt autoremove && sudo apt update && sudo apt install -y apt-transport-https curl

sudo curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -

# 在/etc/apt/sources.list.d/kubernetes.list中添加aliyun的镜像地址

sudo tee /etc/apt/sources.list.d/kubernetes.list <<-'EOF'
deb https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main
EOF

sudo apt update


apt-cache madison kubeadm

# 安装 kubelet kubeadm kubectl
sudo apt install -y kubelet kubeadm kubectl

# 阻止apt-upgrade时更新安装包
sudo apt-mark hold kubelet kubeadm kubectl

# 查看 kubelet版本 和 kubeadm版本
kubelet --version
#Kubernetes v1.18.2
kubeadm version

把上面的步骤改写成 playbook 的形式：

sudo vim install_k8s.yml

- hosts: servers_all
  become: true

  tasks:
    - name: install apt-transport-https curl
      shell: apt autoremove && apt update && apt install -y apt-transport-https curl

    - name: add apt-key
      shell: curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg |  apt-key add -


    - name: add apt to kubernetes.list
      shell: echo "deb https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main" >/etc/apt/sources.list.d/kubernetes.list

    - name: apt update
      shell: apt update

    - name: install kubectl
      shell: apt install kubectl -y
    - name: install kubelet
      shell: apt install kubelet -y
    - name: install kubeadm
      shell: apt install kubeadm -y
    - name: hold update
      shell: apt-mark hold kubelet kubeadm kubectl

ansible-playbook install_k8s.yml

验证是否安装成功：

ansible servers_all -m shell -a "kubelet --version"
ansible servers_all -m shell -a "kubeadm version"
ansible servers_all -m shell -a "kubectl version"

Ansible 执行 k8s 集群的配置与初始化

---

手动配置步骤：

1、在 k8s-master 上

# 查看 k8s 集群需要安装的镜像
kubeadm config images list --kubernetes-version=v1.18.0

# 将 etc/docker/daemon.json 插入 exec-opts": ["native.cgroupdriver=systemd"] ，上一篇已提前设置好

# 初始化 k8s-master
sudo kubeadm init \
		--kubernetes-version=1.18.0  \
		--apiserver-advertise-address=120.xxx.xx.174  \
		--image-repository registry.aliyuncs.com/google_containers  \
		--service-cidr=10.10.0.0/16 \
		--pod-network-cidr=10.122.0.0/16

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config


# 安装网络组件 calico
kubectl apply -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml

2、在 k8s-nodes 上

# 创建 token（有效期24h）
kubeadm token create

# 获取 token
kubeadm token list

# 获取 discovery-token-ca-cert-hash （不会变）
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

# 将 该k8s-node 加入集群
kubeadm join 120.xxx.xx.174:6443 --token 1jyhoi.x8yvhnfizayfjn03 \
    --discovery-token-ca-cert-hash sha256:xxx

如果此前有加入过，或者 k8s-node 修改过 hostname，则需要移除并重新加入

# 在 master执行
kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>

# 在需要被删除节点的机器执行
kubeadm reset

3、 查看是否安装成功

kubectl get node

kubectl get pod --all-namespaces

kubectl get cs

playbook

1、主节点初始化

sudo vim init_k8s.yml

- hosts: k8s-master
  become: true
  gather_facts: False
  tasks:
    - name: init master
      shell: kubeadm init \
        --kubernetes-version=1.19.3  \
        --apiserver-advertise-address=120.133.xx.xxx  \
        --image-repository registry.aliyuncs.com/google_containers  \
        --service-cidr=10.10.0.0/16 \
        --pod-network-cidr=10.122.0.0/16
    - name: active config
      shell: mkdir -p $HOME/.kube && cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && chown $(id -u):$(id -g) $HOME/.kube/config
    - name: install calico
      shell: kubectl apply -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml

ansible-playbook init_k8s.yml

2、从节点加入

sudo vim init_k8s.yml

- hosts: k8s-nodes
  become: true
  gather_facts: False
  tasks:
    - name: join nodes
      shell: kubeadm join 120.xxx.xx.174:6443 --token "abc"    --discovery-token-ca-cert-hash sha256:"abc"

3、如果需要批量删除节点并重新加入

sudo vim reset k8s-nodes.yml

- hosts: k8s-nodes
  become: true
  gather_facts: False
  tasks:
    - name: install expect
      shell: apt install expect -y
    - name: install pexpect
      shell: pip install pexpect
    - name: reset nodes
      expect:
        command: kubeadm reset
        responses:
          (?i)[y/N]: "y"
      shell: kubeadm reset -y

ansible-playbook k8s-nodes.yml