2020-08-08 在国内使用Kubespray搭建Kubernetes集群

前期准备

1. 虚拟机准备

本例涉及5个节点，使用kubespray-2.12.7搭建2个master节点和2个node节点的kubernetes集群，其中node0作为ansible的控制节点，node1-node4是受控节点。

操作系统选用ubuntu18.04，且以下所有操作都是以root用户进行的

节点名称	角色	ip地址
node0	controller	192.168.0.2
node1	kube-master	192.168.0.3
node2	kube-master	192.168.0.4
node3	kube-node	192.168.0.5
node4	kube-node	192.168.0.6

2. 镜像准备

• 在国外租一台服务器，下载谷歌镜像并推送到阿里云镜像仓库，脚本push_img.sh内容如下：

registry_prefix="registry.ap-northeast-1.aliyuncs.com/google-containers-lpz/"

google_images=(
    "gcr.io/google-containers/kube-proxy:v1.16.11"
    "gcr.io/google-containers/kube-controller-manager:v1.16.11"
    "gcr.io/google-containers/kube-scheduler:v1.16.11"
    "gcr.io/google-containers/kube-apiserver:v1.16.11"
    "gcr.io/google-containers/k8s-dns-node-cache:1.15.8"
    "gcr.io/google-containers/cluster-proportional-autoscaler-amd64:1.6.0"
    "gcr.io/google-containers/pause:3.1"
    "gcr.io/google_containers/pause-amd64:3.1"
);

for images in ${google_images[@]}
do
#截取镜像地址的最后一段
container_name=(${images//\// }) 

#拉取镜像并推送到阿里云的镜像仓库
sudo docker pull $images
sudo docker tag $images $registry_prefix${container_name[2]}
sudo docker push $registry_prefix${container_name[2]}

done

• 以下文件推送到本地FTP服务器：

https://storage.googleapis.com/kubernetes-release/release/v1.16.11/bin/linux/amd64/kubelet

https://storage.googleapis.com/kubernetes-release/release/v1.16.11/bin/linux/amd64/kubectl

https://storage.googleapis.com/kubernetes-release/release/v1.16.11/bin/linux/amd64/kubeadm

https://github.com/coreos/etcd/releases/download/v3.3.12/etcd-v3.3.12-linux-amd64.tar.gz

https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz

https://github.com/projectcalico/calicoctl/releases/download/v3.7.3/calicoctl-linux-amd64

3. 镜像修改

• 修改kubespray-2.12.7/roles/download/defaults/main.yml

由于dashboard-1.10.1和kubernetesv1.16不兼容，需要将dashboard的镜像地址修改为kubernetesui/dashboard:v2.0.0-beta8

#gcr_image_repo: "gcr.io"
-> gcr_image_repo: "registry.ap-northeast-1.aliyuncs.com"
#kube_image_repo: "{{ gcr_image_repo }}/google-containers"
-> kube_image_repo: "{{ gcr_image_repo }}/google-containers-lpz"

#pod_infra_image_repo: "{{ gcr_image_repo }}/google_containers/pause-{{ image_arch }}"
-> pod_infra_image_repo: "{{ kube_image_repo }}/pause-{{ image_arch }}"

#dashboard_image_repo: "{{ gcr_image_repo }}/google_containers/kubernetes-dashboard-{{ image_arch }}"
-> dashboard_image_repo: "kubernetesui/dashboard"
#dashboard_image_tag: "xxx"
-> dashboard_image_tag: "v2.0.0-beta8"

#新增ftp服务器地址
-> ftp_url: ftp://192.168.0.2
#kubelet_download_url: "https://storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubelet"
-> kubelet_download_url: "{{ ftp_url }}/kubelet"
#kubectl_download_url: "https://storage.googleapis.com/kubernetes-release/release/{{ kube_version }}/bin/linux/{{ image_arch }}/kubectl"
-> kubectl_download_url: "{{ ftp_url }}/kubectl"
#kubeadm_download_url: "https://storage.googleapis.com/kubernetes-release/release/{{ kubeadm_version }}/bin/linux/{{ image_arch }}/kubeadm"
-> kubeadm_download_url: "{{ ftp_url }}/kubeadm"
etc#d_download_url: "https://github.com/coreos/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"
-> etcd_download_url: "{{ ftp_url }}/etcd-v3.3.12-linux-amd64.tar.gz"
#cni_download_url: "https://github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
-> cni_download_url: "{{ ftp_url }}/cni-plugins-linux-amd64-v0.8.6.tgz"
#calicoctl_download_url: "https://github.com/projectcalico/calicoctl/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
-> calicoctl_download_url: "{{ ftp_url }}/calicoctl-linux-amd64"

• 修改kubespray-2.12.7/inventory/sample/group_vars/k8s-cluster/k8s-cluster.yml

#kube_image_repo: "{{ gcr_image_repo }}/google-containers"
-> kube_image_repo: "{{ gcr_image_repo }}/google-containers-lpz"

安装过程

1.docker的安装和配置

(1) 安装指定版本的docker

提前下载docker，配置阿里云镜像加速器，以从阿里云镜像仓库拉取镜像。

kubespray-2.12.7对应的docker版本为 docker-ce=5:18.09.7_3-0ubuntu-bionic

• 添加Docker官方的GPG密钥

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

• 设置statble存储库

$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" 

• 安装docker-ce=5:18.09.7_3-0ubuntu-bionic

$ sudo apt update
$ sudo apt install -y docker-ce=5:18.09.7~3-0~ubuntu-bionic

(2) 配置docker镜像加速器

$ tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://htdsx47m.mirror.aliyuncs.com"]
}
EOF
$ systemctl daemon-reload
$ systemctl restart docker

2. 使用Kubespray部署Kubernetes集群

(1) 配置控制节点对其他节点的SSH免密登录

• 在controller节点生成密钥对(位于~/.ssh/)

$ ssh-keygen -t rsa

• 将公钥上传到其他节点

在上传之前要确保节点的ssh允许root用户登录和密码登录（PermitRootLogin和PasswordAuthentication字段为yes）

$ ssh-copy-id -i .ssh/id_rsa.pub root@192.168.0.3
$ ssh-copy-id -i .ssh/id_rsa.pub root@192.168.0.4
$ ssh-copy-id -i .ssh/id_rsa.pub root@192.168.0.5
$ ssh-copy-id -i .ssh/id_rsa.pub root@192.168.0.6

• 验证是否成功,如果可以直接登录则成功

$ ssh ubuntu@192.168.0.3

(2) 在控制节点上下载、安装kubepray

• 安装pip3

$ sudo apt install -y python3-pip

• 配置pip3加速

$ mkdir ~/.pip
$ cat > ~/.pip/pip.conf << EOF
[global]
trusted-host=mirrors.aliyun.com
index-url=http://mirrors.aliyun.com/pypi/simple/
EOF

• 下载kubespray并解压

$ cd ~
$ wget https://github.com/kubernetes-sigs/kubespray/archive/v2.12.7.zip
$ unzip v2.12.7.zip

• 安装kubespray，部署kubernetes

$ cd kubespray-2.12.7/
$ sudo pip3 install -r requirements.txt
$ cp -rfp inventory/sample inventory/mycluster
$ declare -a IPS=(192.168.0.3 192.168.0.4 192.168.0.5 192.168.0.6)
$ CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

• 修改hosts.ini的内容

$ vi inventory/mycluster/hosts.yaml

• 最后，执行安装过程。ansible用到的python为python2

$ ansible-playbook -i inventory/mycluster/hosts.yaml  --become --become-user=root cluster.yml

Reference ：
[1]: https://blog.csdn.net/liqi_q/article/details/83030737
[2]: https://www.cnblogs.com/ltxdzh/p/10113194.html#_label2
[3]: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/getting-started.md