手把手教你搭建K3S产线集群

简介

K3S是Kubernetes(K8S)的精简版，官方说是去除了一些云厂商相关的东西。反正在国内也使用不了有的云厂商。部署也比K8S简单了许多。我自己使用了一段时间，感觉还不错。如果你想部署私有化的K8S集群，不妨尝试一下K3S。接下来一步一步教你部署一个产线级别的高可用集群。

资源准备

1. 虚拟机6台(2 Master+4Worker),如果要高可用至少要2个Master

IP	角色	说明
10.128.142.216	Ansible	用于初始化阶节点
10.128.148.141	Master-1	K3S里面master也叫server
10.128.148.165	Master-2
10.128.148.240	Worker-1	K3S里面worker也叫agent
10.128.148.150	Worker-2
10.128.148.214	Worker-3
10.128.148.237	Worker-4

 

2. Mysql数据一个(K3S，可用使用sqlite3,etcd3,mysql, Postgres等数据库作为后端存储，我选用使用比较多的Mysql)

IP: 10.128.97.77

端口: 35474

数据库: k3s

用户: k3s

密码: yourpassword

3. SSL证书一个

4. 负载均衡一个( IP : 10.128.148.28），用于代理两个master节点的6443端口，实现K3S API server的高可用。

架构图

部署步骤

1. 初始化所有节点，由于机器比较多，所以我使用ansible来进行初始化。找一台另外的主机安装ansible,然后与所有的6台主机做ssh的免密，这些步骤我就不写了。如果ansible的使用方法你不知道，需要自己去补一下。

     a. 将所有需要的文件拷贝到Ansible的这台服务器上面
       

- k3s (https://github.com/rancher/k3s/releases/download/v1.20.2+k3s1/k3s)
- k3s-airgap-images-amd64.tar (https://github.com/k3s-io/k3s/releases/download/v1.20.2%2Bk3s1/k3s-airgap-images-amd64.tar)
- install.sh  (下载地址https://get.k3s.io/)
- registries.yaml (用于containerd加速拉取镜像的,K3S的最新版本已经抛弃了docker)

   registries.yaml文件内容

mirrors:
  "docker.io":
    endpoint:
      - "https://fogjl973.mirror.aliyuncs.com"
      - "https://registry-1.docker.io"

b. 然后在同级目录执行初始化ansible脚本,不要遗漏了命令最后的"," 逗号。

ansible-playbook  init-k3s-node.yaml  -i "10.128.148.240,10.128.148.237,10.128.148.214,10.128.148.150,10.128.148.141,10.128.148.165,"

init-k3s-node.yaml

---
- hosts: all
  gather_facts: no
  tasks:
  - name: copy install.sh
    copy: src=install.sh dest=/usr/local/bin/install.sh mode=0775
  - name: copy k3s
    copy: src=k3s  dest=/usr/local/bin/k3s mode=0755
  - name: create dir
    file: state=directory path=/var/lib/rancher/k3s/agent/images
  - name: copy images
    copy: src=k3s-airgap-images-amd64.tar dest=/var/lib/rancher/k3s/agent/images
  - name: create agent dir
    file: state=directory path=/data/k3s/agent
  - name: create k3s dir
    file: state=directory path=/etc/rancher/k3s/
  - name: copy registries.yaml
    copy: src=registries.yaml dest=/etc/rancher/k3s/registries.yaml
  - name: install nfs-utils
    yum: name=nfs-utils state=latest
  - name: enable rpcbind
    service: name=rpcbind state=started enabled=true

2. 登录Master-1节点,然后执行以下命令，tls-san是添加的SLB的地址，这个参数可以多次使用，如果你有多个IP或者有多个域名可以多次添加。

INSTALL_K3S_SKIP_DOWNLOAD=true install.sh server \
  --datastore-endpoint='mysql://k3s:yourpassword@tcp(10.128.97.77:35474)/k3s' \
  --node-taint CriticalAddonsOnly=true:NoExecute \
  --tls-san 10.128.148.28 \
  --tls-san k3s.yourdomain.com \
  --data-dir /data/k3s \
  --node-name master-10-128-148-141 \
  --with-node-id master-10-128-148-141

当命令完成以后，你就可以执行kubectl get node 命令查看刚刚创建的节点

[root@master-1 ]# kubectl get node
NAME                             STATUS   ROLES                  AGE     VERSION
master-10-128-148-141-d49b6858   Ready    control-plane,master   3d15h   v1.20.2+k3s1

3. 登录Master-2节点，然后执行以下命令

INSTALL_K3S_SKIP_DOWNLOAD=true install.sh server \
  --datastore-endpoint='mysql://k3s:yourpassword@tcp(10.128.97.77:35474)/k3s' \
  --node-taint CriticalAddonsOnly=true:NoExecute \
  --tls-san 10.128.148.28 \
  --tls-san k3s.yourdomain.com \
  --data-dir /data/k3s \
  --node-name master-10-128-148-165 \
  --with-node-id master-10-128-148-165

4. 将Master-1节点生成的token文件(路径: /var/lib/rancher/k3s/server/node-token)，复制到Ansible服务器中，然后批量的推送到所有的worker节点中。这个token是worker服务器加入时的凭据。

ansible-playbook push_token.yaml -i "10.128.148.240,10.128.148.237,10.128.148.214,10.128.148.150,"

push_token.yaml文件内容

---
- hosts: all
  gather_facts: no
  tasks:
  - name: create agent dir
    file: state=directory path=/data/k3s/agent
  - name: copy token
    copy: src=node-token dest=/data/k3s/agent/node-token

5. 登录worker节点，然后执行以下的命令。

worker-1

    INSTALL_K3S_SKIP_DOWNLOAD=true K3S_URL=https://k3s.yourdomain.com:6443 K3S_TOKEN_FILE=/data/k3s/agent/node-token install.sh agent \
  --data-dir /data/k3s \
  --node-name worker-10-128-148-240 \
  --with-node-id worker-10-128-148-240

worker-2

    INSTALL_K3S_SKIP_DOWNLOAD=true K3S_URL=https://k3s.yourdomain.com:6443 K3S_TOKEN_FILE=/data/k3s/agent/node-token install.sh agent \
  --data-dir /data/k3s \
  --node-name worker-10-128-148-150 \
  --with-node-id worker-10-128-148-150

worker-3

  INSTALL_K3S_SKIP_DOWNLOAD=true K3S_URL=https://k3s.yourdomain.com:6443 K3S_TOKEN_FILE=/data/k3s/agent/node-token install.sh agent \
  --data-dir /data/k3s \
  --node-name worker-10-128-148-214 \
  --with-node-id worker-10-128-148-214

worker-4

    INSTALL_K3S_SKIP_DOWNLOAD=true K3S_URL=https://k3s.yourdomain.com:6443 K3S_TOKEN_FILE=/data/k3s/agent/node-token install.sh agent \
  --data-dir /data/k3s \
  --node-name worker-10-128-148-237 \
  --with-node-id worker-10-128-148-237

6. 等所有的worker节点加入到集群中以后，就可以在Master-1节点上，查看所有的node了

[root@master-1 ]# kubectl get node
NAME                             STATUS   ROLES                  AGE     VERSION
worker-10-128-148-214-14d34aaf   Ready    <none>                 3d15h   v1.20.2+k3s1
master-10-128-148-165-c6f01320   Ready    control-plane,master   3d16h   v1.20.2+k3s1
worker-10-128-148-237-41ed02e3   Ready    <none>                 3d15h   v1.20.2+k3s1
worker-10-128-148-240-409e6e51   Ready    <none>                 3d15h   v1.20.2+k3s1
master-10-128-148-141-d49b6858   Ready    control-plane,master   3d15h   v1.20.2+k3s1
worker-10-128-148-150-a01bd0da   Ready    <none>                 3d15h   v1.20.2+k3s1

7.Master-1节点上的/etc/rancher/k3s/k3s.yaml 文件，是K3S的配置文件(同K8S config文件).拷贝到其它的主机上一样可以使用，只需将里面的地址修改为tls-san的地址。

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: 
.....
    server: https://k3s.yourdomain.com:6443 #修改这里
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
......

在K3S中部署kubernetes-dashbord，请参考以下文章:

https://blog.csdn.net/lwlfox/article/details/113403133