如何在VirtualBox的CentOS虚拟机中安装阿里云版本Minikube

一、虚拟机启用嵌套VT-x/AMD-V

嵌套虚拟化特性在VirtualBox虚拟机中默认是不启用的（设置-系统-处理器）：

 打开Windows Powershell，进入VirtualBox安装目录，将要安装minikube的虚拟机启用嵌套VT-x/AMD-V。

# 进入安装目录
cd 'C:\Program Files\Oracle\VirtualBox\'

# 列出所有虚拟机
.\VBoxManage.exe list vms
"测试机1" {12027609-b5b5-4364-8ea5-5cc7d4aa6064}
"测试机2" {bdd664c6-4292-4c05-89ff-c09219bf0bfa}
"测试机3" {6e83cf63-597b-4c03-8836-11311deb04ac}


# 打开嵌套虚拟化功能
.\VBoxManage.exe modifyvm "测试机1" --nested-hw-virt on

启用完成后可以看到界面中该选项已勾选：

 二、安装依赖组件

安装docker

yum install docker -y
systemctl enable docker
systemctl start docker

添加阿里云kubenetes yum源

# /etc/yum.repos.d/kubenetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

# 生成元数据缓存
yum makecache

安装kubectl、kubelet

yum install kubectl -y
yum install kubelet -y
systemctl enable kubelet

三、关闭虚拟机swap、selinux、firewalld

# 临时关闭swap
swapoff -a

# 临时关闭selinux，如永久关闭请配置为permissive
setenforce 0

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

永久关闭swap可注释掉/etc/fstab中的swap行，然后重启。永久关闭selinux可编辑/etc/sysconfig/selinux，配置为SELINUX=permissive，然后重启。此处为常规操作不详述。

四、编辑虚拟机hosts文件

与安装k8s类似，需要添加主机名解析

echo "127.0.0.1 test1" >> /etc/hosts

其中test1为虚拟机主机名。

如果不添加该解析，启动minikube时会有如下报错：

[WARNING Hostname]: hostname "test1" could not be reached
[WARNING Hostname]: hostname "test1": lookup test1 on 172.18.3.4:53: no such host

五、安装minikube

使用阿里云定制版，可访问阿里云国内镜像，解决无法访问海外资源的问题。

curl -Lo minikube https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.18.1/minikube-linux-amd64
chmod +x minikube
mv minikube /usr/local/bin/

六、启动minikube

安装conntrack（后面使用–driver=none启动，依赖此包）

yum install conntrack -y

使用如下命令启动minikube

minikube start --registry-mirror="https://na8xypxe.mirror.aliyuncs.com" --driver=none

使用–driver=none的好处是可以直接使用root运行minikube，无需再配置其他用户。缺点是安全性降低、稳定性降低、数据丢失风险、无法使用–cpus、–memory进行资源限制等等，但这不是我们需要考虑的，因为本身安装minikube就是测试学习用的。关于driver的选择，详细可以参看：none | minikube (k8s.io)

启动时我们看到如下报错：

stderr:
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

根据提示进行解决即可：

echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

再次尝试启动，启动成功：

[root@test1 ~]# minikube start --registry-mirror="https://na8xypxe.mirror.aliyuncs.com" --driver=none
* minikube v1.18.1 on Centos 7.6.1810
* Using the none driver based on existing profile
* Starting control plane node minikube in cluster minikube
* Restarting existing none bare metal machine for "minikube" ...
* OS release is CentOS Linux 7 (Core)
* Preparing Kubernetes v1.20.2 on Docker 1.13.1 ...
  - Generating certificates and keys ...
  - Booting up control plane ...
  - Configuring RBAC rules ...
* Configuring local host environment ...
* 
! The 'none' driver is designed for experts who need to integrate with an existing VM
* Most users should use the newer 'docker' driver instead, which does not require root!
* For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/
* 
! kubectl and minikube configuration will be stored in /root
! To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:
* 
  - sudo mv /root/.kube /root/.minikube $HOME
  - sudo chown -R $USER $HOME/.kube $HOME/.minikube
* 
* This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
* Verifying Kubernetes components...
  - Using image registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner:v4 (global image repository)
* Enabled addons: storage-provisioner, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

七、从宿主机访问minikube dashboard

使用如下命令启动dashboard：

[root@test1 ~]# minikube dashboard
* Verifying dashboard health ...
* Launching proxy ...
* Verifying proxy health ...
http://127.0.0.1:33088/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/

使用ctrl+c可关闭。从上面输出的信息可以看到，dashboard绑定的IP地址为本地回环地址127.0.0.1，这意味着该地址只能在本地访问。如果虚拟机是没有GUI的，那么如何从宿主机也就是我们的Windows中访问dashborad呢？可以进行如下操作：

# 后台运行dashboard
[root@test1 ~]# nohup minikube dashboard &
[1] 19609

# 使用proxy代理到虚拟机的指定端口8011和地址0.0.0.0(表示所有地址)
[root@test1 ~]# kubectl proxy --port=8011 --address 0.0.0.0
Starting to serve on [::]:8011

然后我们在VirtualBox中将这台虚拟机的8011端口NAT到宿主机的8011端口（设置-网络），配置方法如下：

 将网卡的连接方式改为NAT，端口转发中配置转发规则，其中子系统IP端口即表示虚拟机的IP端口。

配置完成后，即可在宿主机中访问dashboard，将本段落开头的dashboard url中的IP替换为localhost/127.0.0.1（此处表示宿主机IP），端口替换为8011，即：http://127.0.0.1:8011/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/，打开浏览器可访问：