minikube安装笔记

官方文档：Welcome! | minikube

环境：centos7、docker-v23.0.2、kubernetes-v1.30.0、minikube-v1.33.1。

增加docker和k8s的yum源：

docker-ce.repo：

wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

docker配置/etc/docker/daemon.json：

{
  "exec-opts": [
    "native.cgroupdriver=cgroupfs"
  ],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "registry-mirrors": [
    "https://8xpk5wnt.mirror.aliyuncs.com"
  ]
}

然后systemctl restart docker。

kubernetes.repo：

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0

安装minikube：

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

安装kubectl：

minikube kubectl
sudo cp ~/.minikube/cache/linux/amd64/*/kubectl /usr/local/bin/

kubectl命令自动补全：

yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo 'source <(kubectl completion bash)' >> ~/.bashrc

docker命令自动补全：

curl -L https://raw.githubusercontent.com/docker/cli/v23.0.2/contrib/completion/bash/docker -o ~/.bash_docker_completion
source ~/.bash_docker_completion
echo 'source ~/.bash_docker_completion' >> ~/.bashrc

采用宿主驱动，单节点启动命令：

minikube start --image-mirror-country='cn' -v=6 --kubernetes-version='v1.30.0' --driver=none --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

采用docker驱动，单节点启动命令：

minikube start --memory=2200mb --image-mirror-country='cn' --kubernetes-version='v1.30.0' --driver=docker --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

采用docker驱动，多节点启动命令：

minikube start --memory=2200mb --image-mirror-country='cn' --kubernetes-version='v1.30.0' --driver=docker --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --nodes 2 -p multinode-demo

示例：

[work@el7 minikube]$ cat minikube-start-nodes.sh
minikube start --memory=2200mb --image-mirror-country='cn' --kubernetes-version='v1.30.0' --driver=docker --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --nodes 2 -p multinode-demo

[work@el7 minikube]$ sh minikube-start-nodes.sh
😄  [multinode-demo] minikube v1.33.1 on Centos 7.9.2009
✨  Using the docker driver based on user configuration
✅  Using image repository registry.cn-hangzhou.aliyuncs.com/google_containers
📌  Using Docker driver with root privileges
👍  Starting "multinode-demo" primary control-plane node in "multinode-demo" cluster
🚜  Pulling base image v0.0.44 ...
❗  minikube was unable to download registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.44, but successfully downloaded docker.io/kicbase/stable:v0.0.44 as a fallback image
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...
🐳  Preparing Kubernetes v1.30.0 on Docker 26.1.1 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass

👍  Starting "multinode-demo-m02" worker node in "multinode-demo" cluster
🚜  Pulling base image v0.0.44 ...
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.49.2
🐳  Preparing Kubernetes v1.30.0 on Docker 26.1.1 ...
    ▪ env NO_PROXY=192.168.49.2
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "multinode-demo" cluster and "default" namespace by default
[work@el7 minikube]$

[work@el7 minikube]$ kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS    RESTARTS        AGE
kube-system   coredns-7c445c467-7qrf8                  1/1     Running   2 (5m3s ago)    5m39s
kube-system   etcd-multinode-demo                      1/1     Running   0               5m53s
kube-system   kindnet-4v4c6                            1/1     Running   0               5m1s
kube-system   kindnet-522td                            1/1     Running   0               5m40s
kube-system   kube-apiserver-multinode-demo            1/1     Running   0               5m53s
kube-system   kube-controller-manager-multinode-demo   1/1     Running   0               5m57s
kube-system   kube-proxy-kkn5t                         1/1     Running   0               5m1s
kube-system   kube-proxy-n76jj                         1/1     Running   0               5m40s
kube-system   kube-scheduler-multinode-demo            1/1     Running   0               5m57s
kube-system   storage-provisioner                      1/1     Running   2 (5m31s ago)   5m49s
[work@el7 minikube]$

开启kube-proxy的ipvs模式：

#编辑 kube-proxy 配置以启用 ipvs 模式。您可以使用以下命令编辑配置：
kubectl edit configmap kube-proxy -n kube-system
#在配置中，找到 mode 字段并将其设置为 ipvs。
#mode: "ipvs"
#重新启动 kube-proxy 以应用新配置：
kubectl rollout restart daemonset kube-proxy -n kube-system
#验证 ipvs 模式是否已启用。您可以使用以下命令检查 kube-proxy 的日志：
kubectl -n kube-system logs kube-proxy-*
#日志中应该包含以下行：
#Using ipvs Proxier

添加nginx服务：

进入node容器内，下载镜像：

渡渡鸟镜像同步站

docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/nginx:1.27.0
docker tag  swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/nginx:1.27.0  docker.io/nginx:1.27.0

或者通过minikube命令添加镜像：

[work@el7 minikube]$ minikube profile list
|----------------|-----------|---------|--------------|------|---------|---------|-------|----------------|--------------------|
|    Profile     | VM Driver | Runtime |      IP      | Port | Version | Status  | Nodes | Active Profile | Active Kubecontext |
|----------------|-----------|---------|--------------|------|---------|---------|-------|----------------|--------------------|
| multinode-demo | docker    | docker  | 192.168.49.2 | 8443 | v1.30.0 | Running |     2 |                | *                  |
|----------------|-----------|---------|--------------|------|---------|---------|-------|----------------|--------------------|

[work@el7 minikube]$  minikube image list -p  multinode-demo
registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner:v5
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kindnetd:v20240202-8f1494ea
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.12-0
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.11.1

[work@el7 minikube]$ minikube image load nginx_1.27.0.tar -p multinode-demo

[work@el7 minikube]$
[work@el7 minikube]$ minikube image list -p  multinode-demo
registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner:v5
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kindnetd:v20240202-8f1494ea
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.12-0
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.11.1
docker.io/library/nginx:1.27.0
[work@el7 minikube]$

Deployment配置：nginx-deployment-2pod.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 100%
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: docker.io/nginx:1.27.0
          imagePullPolicy: Never
          resources:
            requests:
              memory: "50Mi"
            limits:
              memory: "200Mi"
          ports:
            - name: http
              containerPort: 80
      terminationGracePeriodSeconds: 6

Service配置： nginx-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - protocol: TCP
      nodePort: 30080
      port: 8080
      targetPort: 80
  externalTrafficPolicy: Local

部署服务：

kubectl apply -f nginx-deployment-2pod.yaml
kubectl apply -f nginx-svc.yaml

------

遇到的错误1，kubelet启动pod失败：

[kubelet-check] Initial timeout of 40s passed.
This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration

查看kubelet日志：

sudo tail -f /var/log/messages | grep kubelet

如下：

kuberuntime_manager.go:782] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed pulling image \"registry.k8s.io/pause:3.6\": Error response from daemon: Head \"https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.6\": dial tcp 74.125.23.82:443: connect: connection refused" pod="kube-system/kube-controller-manager-control-plane.minikube.internal"

解决办法：

docker pull k8simage/pause:3.6
id=`docker images -q k8simage/pause:3.6`
docker tag $id registry.k8s.io/pause:3.6

遇到的错误2，kubectl非root账号执行报错：

执行 kubectl get pod 显示：error: unable to read client-key: permission denied

解决办法：

sudo cp -r /root/.minikube/ $HOME/.minikube/                  
sudo chown $USER -R $HOME/.minikube/

再次执行 kubectl get pod 显示：x509: certificate signed by unknown authority

解决办法：

sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config

遇到的错误3，node状态NotReady：

$ kubectl get nodes
NAME                              STATUS     ROLES    AGE   VERSION
control-plane.minikube.internal   NotReady   <none>   28m   v1.26.1

查看kubelet日志：kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"

解决办法：

#安装bridge、vlan、macvlan、tuning等二进制插件
curl -O -L https://github.com/containernetworking/plugins/releases/download/v1.2.0/cni-plugins-linux-amd64-v1.2.0.tgz
mkdir -p /opt/cni/bin
tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-v1.2.0.tgz

遇到的错误4，minikube重启报错certificate apiserver not signed：

error execution phase certs/apiserver: [certs] certificate apiserver not signed by CA certificate ca: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "minikubeCA")

解决办法：

#重来一遍
minikube stop
minikube delete
minikube start --image-mirror-country='cn' --kubernetes-version='v1.26.1' --driver=none --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

 遇到的错误5，The 'none driver does not support multiple profiles

当试图启动多个node时：

minikube start --image-mirror-country='cn' -v=5 --kubernetes-version='v1.26.1' --driver=none --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --nodes 2 -p multinode-demo

报错：X Exiting due to DRV_UNSUPPORTED_PROFILE: The 'none driver does not support multiple profiles: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

错误6：拉取镜像失败

#示例
docker pull gcr.io/k8s-minikube/kicbase:v0.0.44
#改为
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.44
#docker images查询IMAGE ID
docker tag {IMAGE ID} gcr.io/k8s-minikube/kicbase:v0.0.44

错误7：permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

解决办法：

sudo usermod -aG docker $USER && newgrp docker

错误8：ErrImageNeverPull: Container image "xxxx:vxx" is not present with pull policy of Never

解决办法：

使用Minikube的本地Docker镜像-CSDN博客

错误9：因 RUNTIME_ENABLE 错误而退出：Failed to start container runtime: Temporary Error: sudo /usr/bin/crictl version

解决办法：

查看docker启动参数：

/usr/bin/dockerd -H unix:///var/run/docker.sock --containerd=/run/containerd/containerd.sock

为minikube设定参数：--cri-socket='/run/containerd/containerd.sock'

--container-runtime=docker --driver=docker --cri-socket='/run/containerd/containerd.sock'

错误10：Failed to connect to bus: No such file or directory

解决办法：

docker@minikube:~$ systemctl restart docker
Failed to connect to bus: No such file or directory
docker@minikube:~$ sudo -s
root@minikube:/home/docker# 
root@minikube:/home/docker# systemctl restart docker

错误11：node status is NotReady

[work@el7 k8s]$ kubectl get nodes
NAME                 STATUS     ROLES           AGE     VERSION
multinode-demo       Ready      control-plane   2d20h   v1.30.0
multinode-demo-m02   NotReady   <none>          2d20h   v1.30.0

解决办法：

[work@el7 k8s]$ kubectl describe node multinode-demo-m02
Name:               multinode-demo-m02
Roles:              <none>
Conditions:
  Type             Status    LastHeartbeatTime                 LastTransitionTime                Reason              Message
  ----             ------    -----------------                 ------------------                ------              -------
  Ready            Unknown   Sat, 03 Aug 2024 00:22:53 +0800   Mon, 05 Aug 2024 01:38:28 +0800   NodeStatusUnknown   Kubelet stopped posting node status.

[work@el7 k8s]$ docker exec -it c5f7c9d87677 bash
root@multinode-demo-m02:/# systemctl status kubelet
root@multinode-demo-m02:/# systemctl start kubelet
root@multinode-demo-m02:/# systemctl status kubelet

错误12：apt install ipvsadm: E: Unable to locate package ipvsadm

解决办法：

apt update
apt install ipvsadm

错误13：cannot open '/var/log/message' for reading: No such file or directory

容器内通过systemd-journald记录的系统日志，日志文件默认存放目录是：

/run/log/journal/

查看办法：

journalctl -n 30 -f

错误14：container init: open /proc/sys/net/ipv4/ip_unprivileged_port_start: no such file or directory: unknown

原因：容器里的容器启动了80端口，被限制。

解决办法（要求linux内核版本不低于4.11）：

#临时生效
sysctl net.ipv4.ip_unprivileged_port_start=0
#永久生效
echo "net.ipv4.ip_unprivileged_port_start=0" >> /etc/sysctl.conf

错误15：Exiting due to SVC_URL_TIMEOUT: http://127.0.0.1:43480/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ is not accessible: Temporary Error: unexpected response code: 503

解决办法：

#打印更多日志
minikube dashboard --alsologtostderr -p multinode-demo

#查看dashboard的pod状态
kubectl get pods -A

#查看dashboard的pod详情
kubectl describe pod kubernetes-dashboard-779776cb65-kbklx -n kubernetes-dashboard

显示错误：

  Warning  Failed   7m40s (x20 over 173m)    kubelet  (combined from similar events): Failed to pull image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93": error pulling image configuration: download failed after attempts=6: dial tcp 69.171.247.32:443: i/o timeout
  Normal   BackOff  5m16s (x436 over 3h36m)  kubelet  Back-off pulling image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93"

更换国内镜像：

#拉取国内镜像
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/kubernetesui/dashboard:v2.7.0

#修改tag
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/kubernetesui/dashboard:v2.7.0 docker.io/kubernetesui/dashboard:v2.7.0

#加载到minikube
docker save -o dashboard_v2.7.0.tar kubernetesui/dashboard:v2.7.0
minikube image load dashboard_v2.7.0.tar -p multinode-demo

#删除旧pod重新构建
kubectl delete pod kubernetes-dashboard-779776cb65-kbklx -n kubernetes-dashboard
minikube dashboard --alsologtostderr -p multinode-demo

终极大法从头再来：

rm -drf ~/.minikube/

多节点支持相关文档：

https://minikube.kubernetes.ac.cn/docs/tutorials/multi_node/

CSI 驱动程序和卷快照 | minikube 中文

https://minikube.sigs.k8s.io/docs/drivers/none/

Using Multi-Node Clusters | minikube

Add `local-path-provisioner` addon by presztak · Pull Request #15062 · kubernetes/minikube · GitHub

GitHub - rancher/local-path-provisioner: Dynamically provisioning persistent local storage with Kubernetes