Mac上自动化搭建k8s集群

1.安装部署架构
Mac上搭建三个虚拟机组成k8s集群
2.安装virtualbox
下载地址：https://www.virtualbox.org/wiki/Downloads
3.安装vagrant
下载地址：https://www.vagrantup.com/downloads

//检查安装结果
sunxi@bogon ~ % vagrant version
Installed Version: 2.2.14
Latest Version: 2.2.14
You're running an up-to-date version of Vagrant!
sunxi@bogon ~ % 

4.在浏览器中打开https://app.vagrantup.com/centos/boxes/7/versions/1905.1/providers/virtualbox.box下载box镜像并将其移动至/Users/sunxi/Documents/application/vagrant/box/目录下
5.使用下面命令添加box镜像

vagrant box add --name centos/7 /Users/sunxi/Documents/application/vagrant/box/CentOS-7-x86_64-Vagrant-1905_01.VirtualBox.box

结果显示如下：

sunxi@bogon box % vagrant box add --name centos/7 /Users/sunxi/Documents/application/vagrant/box/CentOS-7-x86_64-Vagrant-1905_01.VirtualBox.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'centos/7' (v0) for provider: 
    box: Unpacking necessary files from: file:///Users/sunxi/Documents/application/vagrant/box/CentOS-7-x86_64-Vagrant-1905_01.VirtualBox.box
==> box: Successfully added box 'centos/7' (v0) for 'virtualbox'!
sunxi@bogon box % 

6.使用下面命令检查添加结果：

vagrant box list

结果如下：

sunxi@bogon box % vagrant box list
centos/7 (virtualbox, 0)
sunxi@bogon box % 

7.搭建k8s集群
（1）创建/Users/sunxi/Documents/application/k8s目录并在该目录下执行命令：

 git clone https://github.com/zhan-yl/kubernetes.git

（2）进入进入kubernetes/vagrant-provisioning目录并打开Vagrantfile文件，该文件定义了1个master和2个worker节点及节点的资源配置，这里可以根据实际情况进行修改

# -*- mode: ruby -*-
# vi: set ft=ruby :

ENV['VAGRANT_NO_PARALLEL'] = 'yes'

Vagrant.configure(2) do |config|

  config.vm.provision "shell", path: "bootstrap.sh"

  # Kubernetes Master Server
  config.vm.define "kmaster" do |kmaster|
    kmaster.vm.box = "centos/7"
    kmaster.vm.hostname = "kmaster.example.com"
    kmaster.vm.network "private_network", ip: "172.42.42.100"
    kmaster.vm.provider "virtualbox" do |v|
      v.name = "kmaster"
      v.memory = 1024
      v.cpus = 1
      # Prevent VirtualBox from interfering with host audio stack
      v.customize ["modifyvm", :id, "--audio", "none"]
    end
    kmaster.vm.provision "shell", path: "bootstrap_kmaster.sh"
  end

  NodeCount = 2

  # Kubernetes Worker Nodes
  (1..NodeCount).each do |i|
    config.vm.define "kworker#{i}" do |workernode|
      workernode.vm.box = "centos/7"
      workernode.vm.hostname = "kworker#{i}.example.com"
      workernode.vm.network "private_network", ip: "172.42.42.10#{i}"
      workernode.vm.provider "virtualbox" do |v|
        v.name = "kworker#{i}"
        v.memory = 2048
        v.cpus = 2
        # Prevent VirtualBox from interfering with host audio stack
        v.customize ["modifyvm", :id, "--audio", "none"]
      end
      workernode.vm.provision "shell", path: "bootstrap_kworker.sh"
    end
  end

end

（3）进入kubernetes/vagrant-provisioning目录执行vagrant up命令完成k8s集群搭建、配置及启动
执行这一步时，曝出如下错误：

sunxi@bogon vagrant-provisioning % vagrant up
Bringing machine 'kmaster' up with 'virtualbox' provider...
Bringing machine 'kworker1' up with 'virtualbox' provider...
Bringing machine 'kworker2' up with 'virtualbox' provider...
==> kmaster: Importing base box 'centos/7'...
==> kmaster: Matching MAC address for NAT networking...
==> kmaster: Setting the name of the VM: kmaster
==> kmaster: Clearing any previously set network interfaces...
==> kmaster: Preparing network interfaces based on configuration...
    kmaster: Adapter 1: nat
    kmaster: Adapter 2: hostonly
==> kmaster: Forwarding ports...
    kmaster: 22 (guest) => 2222 (host) (adapter 1)
==> kmaster: Running 'pre-boot' VM customizations...
==> kmaster: Booting VM...
==> kmaster: Waiting for machine to boot. This may take a few minutes...
    kmaster: SSH address: 127.0.0.1:2222
    kmaster: SSH username: vagrant
    kmaster: SSH auth method: private key
    kmaster: 
    kmaster: Vagrant insecure key detected. Vagrant will automatically replace
    kmaster: this with a newly generated keypair for better security.
    kmaster: 
    kmaster: Inserting generated public key within guest...
    kmaster: Removing insecure key from the guest if it's present...
    kmaster: Key inserted! Disconnecting and reconnecting using new SSH key...
==> kmaster: Machine booted and ready!
==> kmaster: Checking for guest additions in VM...
    kmaster: No guest additions were detected on the base box for this VM! Guest
    kmaster: additions are required for forwarded ports, shared folders, host only
    kmaster: networking, and more. If SSH fails on this machine, please install
    kmaster: the guest additions and repackage the box to continue.
    kmaster: 
    kmaster: This is not an error message; everything may continue to work properly,
    kmaster: in which case you may ignore this message.
==> kmaster: Setting hostname...
==> kmaster: Configuring and enabling network interfaces...
==> kmaster: Rsyncing folder: /Users/sunxi/Documents/application/k8s/kubernetes/vagrant-provisioning/ => /vagrant
==> kmaster: Running provisioner: shell...
    kmaster: Running: /var/folders/cp/_6lr4gqs1bvc70crzvbknk8c0000gn/T/vagrant-shell20210204-82021-apbxe7.sh
    kmaster: [TASK 1] Update /etc/hosts file
    kmaster: [TASK 2] Install docker container engine
    kmaster: [TASK 3] Enable and start docker service
    kmaster: [TASK 4] Disable SELinux
    kmaster: [TASK 5] Stop and Disable firewalld
    kmaster: [TASK 6] Add sysctl settings
    kmaster: [TASK 7] Disable and turn off SWAP
    kmaster: [TASK 8] Add yum repo file for kubernetes
    kmaster: [TASK 9] Install Kubernetes (kubeadm, kubelet and kubectl)
    kmaster: [TASK 10] Enable and start kubelet service
    kmaster: [TASK 11] Enable ssh password authentication
    kmaster: [TASK 12] Set root password
    kmaster: [TASK 13] Set TimeZone
==> kmaster: Running provisioner: shell...
    kmaster: Running: /var/folders/cp/_6lr4gqs1bvc70crzvbknk8c0000gn/T/vagrant-shell20210204-82021-1trwtkz.sh
    kmaster: [TASK 1] Initialize Kubernetes Cluster
    kmaster: [TASK 2] Copy kube admin config to Vagrant user .kube directory
    kmaster: cp: cannot stat ‘/etc/kubernetes/admin.conf’: No such file or directory
    kmaster: [TASK 3] Deploy flannel network
    kmaster: The connection to the server localhost:8080 was refused - did you specify the right host or port?
    kmaster: [TASK 4] Generate and save cluster join command to /joincluster.sh
    kmaster: failed to load admin kubeconfig: open /root/.kube/config: no such file or directory
    kmaster: To see the stack trace of this error execute with --v=5 or higher
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

故障排查：
  首先登录到master节点上确认有没有/etc/kubernetes/admin.conf文件，检查结果没有，所以判断是在执行集群初始化的时候出现的问题，然后在bootstrap_kmaster.sh脚本中找到集群初始化的命令kubeadm init --apiserver-advertise-address=172.42.42.100 --pod-network-cidr=10.244.0.0/16 --image-repository=registry.aliyuncs.com/google_containers --kubernetes-version=v1.18.3并在master节点上单独执行发现以下报错：

  接着调整master节点的资源配置，执行vagrant halt关闭虚拟机集群，通过virtualbox删除master并执行vagrant up重新搭建k8s集群即可。这里需要注意下，我在搭建过程中发现其脚本自动安装的kubelet、kubectl和kubeadm都是最新版本，即1.20.2，所以我将初始化集群命令的参数–kubernetes-version修改为1.20.2与kubelet、kubectl和kubeadm保持一致。

（4）使用vagrant status检查虚拟机运行状态

sunxi@bogon vagrant-provisioning % vagrant status
Current machine states:

kmaster                   running (virtualbox)
kworker1                  running (virtualbox)
kworker2                  running (virtualbox)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

（5）查看集群信息

//登录master节点
vagrant ssh kmaster

[vagrant@kmaster ~]$ kubectl cluster-info
Kubernetes control plane is running at https://172.42.42.100:6443
KubeDNS is running at https://172.42.42.100:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

（6）获取节点状态

[vagrant@kmaster ~]$ kubectl get no
NAME                   STATUS   ROLES                  AGE   VERSION
kmaster.example.com    Ready    control-plane,master   34m   v1.20.2
kworker1.example.com   Ready    <none>                 25m   v1.20.2
kworker2.example.com   Ready    <none>                 22m   v1.20.2

（7）获取系统组件状态

[vagrant@kmaster ~]$ kubectl get -n kube-system all
NAME                                              READY   STATUS    RESTARTS   AGE
pod/coredns-7f89b7bc75-hjjcm                      1/1     Running   0          34m
pod/coredns-7f89b7bc75-vzddj                      1/1     Running   0          34m
pod/etcd-kmaster.example.com                      1/1     Running   0          34m
pod/kube-apiserver-kmaster.example.com            1/1     Running   0          34m
pod/kube-controller-manager-kmaster.example.com   1/1     Running   0          34m
pod/kube-flannel-ds-amd64-7sbmg                   1/1     Running   0          25m
pod/kube-flannel-ds-amd64-c4vw9                   1/1     Running   0          30m
pod/kube-flannel-ds-amd64-dvpdk                   1/1     Running   0          22m
pod/kube-proxy-brkxl                              1/1     Running   0          34m
pod/kube-proxy-dscmw                              1/1     Running   0          25m
pod/kube-proxy-s8xmt                              1/1     Running   0          22m
pod/kube-scheduler-kmaster.example.com            1/1     Running   0          34m

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   34m

NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/kube-flannel-ds-amd64   3         3         3       3            3           <none>                   30m
daemonset.apps/kube-proxy              3         3         3       3            3           kubernetes.io/os=linux   34m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns   2/2     2            2           34m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/coredns-7f89b7bc75   2         2         2       34m

（8）部署nginx应用检查集群安装是否存在问题

[vagrant@kmaster ~]$ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created

//暴露服务端口：
[vagrant@kmaster ~]$ kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed

//检查nginx pod及服务的状态：
[vagrant@kmaster ~]$ kubectl get pod,svc -o wide
NAME                         READY   STATUS    RESTARTS   AGE   IP           NODE                   NOMINATED NODE   READINESS GATES
pod/nginx-6799fc88d8-bm7sg   1/1     Running   0          16m   10.244.2.2   kworker2.example.com   <none>           <none>

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP        51m   <none>
service/nginx        NodePort    10.103.76.79   <none>        80:32399/TCP   16m   app=nginx

//访问nginx服务：
[vagrant@kmaster ~]$ curl kworker2.example.com:32399
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

8.vagrant常见的使用命令：

命令	含义
vagrant up	启动虚拟机
vagrant halt	关闭虚拟机
vagrant status	检查虚拟机状态
vagrant ssh	ssh到虚拟机
vagrant box list	列出所有box列表
vagrant version	获取vagrant版本

注意：整个自动化安装部署脚本中没有添加验证环节，所以在集群搭建完成后需要部署应用验证是否成功