kubespray 离线安装 kubernetes

环境： centos7.6，kubespray-2.14.1，内网配置好 yum 源、pip源、docker-ce 源（可以考虑使用 nexus 代理 ），nexus 代理好 docker hub、aliyun docker registry

1、copy 资产目录

[root@ansible002 kubespray-2.14.1]# cp -r inventory/sample inventory/mycluster

2、配置好内网 pip 源，安装 requirement.txt

[root@ansible002 kubespray-2.14.1]# pip install -r requirements.txt

yum install python-netaddr

3、修改资产文件 inventory/mycluster/inventory.ini

[root@ansible002 kubespray-2.14.1]# cat inventory/mycluster/inventory.ini 
[all]
node1 ansible_host=192.168.1.222
node2 ansible_host=192.168.1.223
node3 ansible_host=192.168.1.224
node4 ansible_host=192.168.1.225
node5 ansible_host=192.168.1.226

# ## configure a bastion host if your nodes are not directly reachable
# bastion ansible_host=x.x.x.x ansible_user=some_user

[kube-master]
node1
node2
node3

[etcd]
node1
node2
node3

[kube-node]
node1
node2
node3
node4
node5

[calico-rr]

[k8s-cluster:children]
kube-master
kube-node
calico-rr

[all:vars]
#ansible_user=gree
#ansible_become=yes
#ansible_user=root
#ansible_ssh_pass=123456
#ansible_sudo_pass=123456

4、更新 docker registry

做好 ansible 服务器到其他服务器的 ssh 免密登陆

[root@ansible002 kubespray-2.14.1]# cat daemon.json 
{
    "insecure-registries": [
        "10.2.7.11"
    ],
    "registry-mirrors": [
        "
https://10.2.7.11"
;
    ]

}

[root@ansible002 kubespray-2.14.1]# ansible all -i inventory/mycluster/inventory.ini -m copy -a "src=daemon.json dest=/etc/docker/daemon.json"

[root@ansible002 kubespray-2.14.1]# ansible all -i inventory/mycluster/inventory.ini -m shell -a "systemctl restart docker"

5、修改变量，主要修改以下几个内容

# Registry overrides
kube_image_repo: "{{ registry_host }}"
gcr_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"

kubeadm_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubeadm"
kubectl_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubectl"
kubelet_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubelet"
# etcd is optional if you **DON'T** use etcd_deployment=host
etcd_download_url: "{{ files_repo }}/kubernetes/etcd/etcd-{{ etcd_version }}-linux-amd64.tar.gz"
cni_download_url: "{{ files_repo }}/kubernetes/cni/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
crictl_download_url: "{{ files_repo }}/kubernetes/cri-tools/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
# If using Calico
calicoctl_download_url: "{{ files_repo }}/kubernetes/calico/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"

# CentOS/Redhat
## Docker
docker_rh_repo_base_url: "{{ yum_repo }}/docker-ce/$releasever/$basearch"
docker_rh_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
## Containerd
extras_rh_repo_base_url: "{{ yum_repo }}/centos/$releasever/extras/$basearch"
extras_rh_repo_gpgkey: "{{ yum_repo }}/containerd/gpg"

# Fedora
## Docker
docker_fedora_repo_base_url: "{{ yum_repo }}/docker-ce/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}"
docker_fedora_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
## Containerd
containerd_fedora_repo_base_url: "{{ yum_repo }}/containerd"
containerd_fedora_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"

# Debian
## Docker
docker_debian_repo_base_url: "{{ debian_repo }}/docker-ce"
docker_debian_repo_gpgkey: "{{ debian_repo }}/docker-ce/gpg"
## Containerd
containerd_debian_repo_base_url: "{{ ubuntu_repo }}/containerd"
containerd_debian_repo_gpgkey: "{{ ubuntu_repo }}/containerd/gpg"
containerd_debian_repo_repokey: 'YOURREPOKEY'

# Ubuntu
## Docker
docker_ubuntu_repo_base_url: "{{ ubuntu_repo }}/docker-ce"
docker_ubuntu_repo_gpgkey: "{{ ubuntu_repo }}/docker-ce/gpg"
## Containerd
containerd_ubuntu_repo_base_url: "{{ ubuntu_repo }}/containerd"
containerd_ubuntu_repo_gpgkey: "{{ ubuntu_repo }}/containerd/gpg"
containerd_ubuntu_repo_repokey: 'YOURREPOKEY'

# If using helm
helm_stable_repo_url: "{{ helm_registry }}"

本文使用的是 centos，docker，所有可以简化为以下

# Registry overrides
kube_image_repo: "192.168.1.12/google_containers"
gcr_image_repo: "192.168.1.12"
docker_image_repo: "192.168.1.12"
quay_image_repo: "192.168.1.12"

kubeadm_download_url: "
http://192.168.1.12/repository/raw-file/kubespray/kubeadm"
;
kubectl_download_url: "
http://192.168.1.12/repository/raw-file/kubespray/kubectl"
;
kubelet_download_url: "
http://192.168.1.12/repository/raw-file/kubespray/kubelet"
;
cni_download_url: "
http://192.168.1.12/repository/raw-file/kubespray/cni-plugins-linux-amd64-v0.8.7.tgz"
;
# etcd is optional if you **DON'T** use etcd_deployment=host
etcd_download_url: "
http://192.168.1.12/repository/raw-file/kubespray/etcd-v3.4.3-linux-amd64.tar.gz"
;
crictl_download_url: "
http://192.168.1.12/repository/raw-file/kubespray/crictl-v1.18.0-linux-amd64.tar.gz"
;
# If using Calico
calicoctl_download_url: "
http://192.168.1.12/repository/raw-file/kubespray/calicoctl-linux-amd64"
;

# CentOS/Redhat
## Docker
docker_rh_repo_base_url: "
http://mirror.gree.com/docker-ce/linux/centos/7/
$basearch/stable"
docker_rh_repo_gpgkey: "
http://mirror.gree.com/docker-ce/linux/centos/gpg"
;


### kube_image_repo image 
pod_infra_image_repo: "192.168.1.12/google_containers/pause"
nodelocaldns_image_repo: "192.168.1.12/google_containers/k8s-dns-node-cache"
dnsautoscaler_image_repo: "192.168.1.12/google_containers/cluster-proportional-autoscaler-amd64"

etcd_deployment_type: host

其中 192.168.1.12 使用 nexus 代理了 docker hub 、aliyun docker registry，是一个 group 类型的 docker registry

docker_rh_repo_base_url 为 docker-ce 源

pod_infra_image_repo、nodelocaldns_image_repo、dnsautoscaler_image_repo 这 3 个 k8s.gcr.io 中的镜像，使用 aliyun docker registry 中 google_containers 仓库中的镜像代替

etcd_deployment_type: host 表示使用二进制安装 etcd

将对应版本的 kubeadm、kubectl、etcd、cni-plugins、calicoctl、crictl 等二进制文件上传到 nexus raw 类型的 repository，以供下载

6、执行 download playbook，查看有还有哪些下载有问题

[root@ansible002 kubespray-2.14.1]# ansible-playbook -i inventory/mycluster/inventory.ini  cluster.yml -t download

7、执行部署命令

[root@ansible002 kubespray-2.14.1]# ansible-playbook -i inventory/mycluster/inventory.ini  cluster.yml

8、遇到的错误与解决方法

错误：

TASK [download : prep_kubeadm_images | Copy kubeadm binary from download dir to system path]
...
"msg": "[Errno 2] No such file or directory"
...

解决方法

必须在远程节点之间启用无密码登陆，或者强制 rsync 使用 ssh

增加参数

- name: prep_kubeadm_images | Copy kubeadm binary from download dir to system path
  synchronize:
    src: "{{ local_release_dir }}/kubeadm-{{ kubeadm_version }}-{{ image_arch }}"
    dest: "{{ bin_dir }}/kubeadm"
    compress: no
    perms: yes
    owner: no
    group: no
    rsync_opts: "-e ssh -p 22"
  delegate_to: "{{ inventory_hostname }}"

错误：

TASK [kubernetes/preinstall : Check that kube_service_addresses is a network range] ***********************************************************************************
fatal: [node1]: FAILED! => {"msg": "The conditional check 'kube_service_addresses | ipaddr('net')' failed. The error was: The ipaddr filter requires python's netaddr be installed on the ansible controller"}

解决方法： 安装 python-netaddr

yum install python-netaddr

参考文章：

https://github.com/kubernetes-sigs/kubespray

https://www.bboy.app/2020/07/20/%E4%BD%BF%E7%94%A8kubespray%E6%90%AD%E5%BB%BA%E7%94%9F%E4%BA%A7%E7%BA%A7%E9%AB%98%E5%8F%AF%E7%94%A8%E9%9B%86%E7%BE%A4/
[https://github.com/kubernetes-sigs/kubespray/blob/master/docs/offline-environment.md](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/offline-environment.md]