官方文档：Welcome! | minikube
环境：centos7、docker-v23.0.2、kubernetes-v1.30.0、minikube-v1.33.1。
增加docker和k8s的yum源：
docker-ce.repo：
wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repodocker配置/etc/docker/daemon.json：
{
"exec-opts": [
"native.cgroupdriver=cgroupfs"
],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"registry-mirrors": [
"https://8xpk5wnt.mirror.aliyuncs.com"
]
}然后systemctl restart docker。
kubernetes.repo：
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0安装minikube：
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube安装kubectl：
minikube kubectl
sudo cp ~/.minikube/cache/linux/amd64/*/kubectl /usr/local/bin/kubectl命令自动补全：
yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo 'source <(kubectl completion bash)' >> ~/.bashrc
docker命令自动补全：
curl -L https://raw.githubusercontent.com/docker/cli/v23.0.2/contrib/completion/bash/docker -o ~/.bash_docker_completion
source ~/.bash_docker_completion
echo 'source ~/.bash_docker_completion' >> ~/.bashrc采用宿主驱动，单节点启动命令：
minikube start --image-mirror-country='cn' -v=6 --kubernetes-version='v1.30.0' --driver=none --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers采用docker驱动，单节点启动命令：
minikube start --memory=2200mb --image-mirror-country='cn' --kubernetes-version='v1.30.0' --driver=docker --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers采用docker驱动，多节点启动命令：
minikube start --memory=2200mb --image-mirror-country='cn' --kubernetes-version='v1.30.0' --driver=docker --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --nodes 2 -p multinode-demo示例：
[work@el7 minikube]$ cat minikube-start-nodes.sh
minikube start --memory=2200mb --image-mirror-country='cn' --kubernetes-version='v1.30.0' --driver=docker --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --nodes 2 -p multinode-demo[work@el7 minikube]$ sh minikube-start-nodes.sh
😄 [multinode-demo] minikube v1.33.1 on Centos 7.9.2009
✨ Using the docker driver based on user configuration
✅ Using image repository registry.cn-hangzhou.aliyuncs.com/google_containers
📌 Using Docker driver with root privileges
👍 Starting "multinode-demo" primary control-plane node in "multinode-demo" cluster
🚜 Pulling base image v0.0.44 ...
❗ minikube was unable to download registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.44, but successfully downloaded docker.io/kicbase/stable:v0.0.44 as a fallback image
🔥 Creating docker container (CPUs=2, Memory=2200MB) ...
🐳 Preparing Kubernetes v1.30.0 on Docker 26.1.1 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔗 Configuring CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass👍 Starting "multinode-demo-m02" worker node in "multinode-demo" cluster
🚜 Pulling base image v0.0.44 ...
🔥 Creating docker container (CPUs=2, Memory=2200MB) ...
🌐 Found network options:
▪ NO_PROXY=192.168.49.2
🐳 Preparing Kubernetes v1.30.0 on Docker 26.1.1 ...
▪ env NO_PROXY=192.168.49.2
🔎 Verifying Kubernetes components...
🏄 Done! kubectl is now configured to use "multinode-demo" cluster and "default" namespace by default
[work@el7 minikube]$[work@el7 minikube]$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-7c445c467-7qrf8 1/1 Running 2 (5m3s ago) 5m39s
kube-system etcd-multinode-demo 1/1 Running 0 5m53s
kube-system kindnet-4v4c6 1/1 Running 0 5m1s
kube-system kindnet-522td 1/1 Running 0 5m40s
kube-system kube-apiserver-multinode-demo 1/1 Running 0 5m53s
kube-system kube-controller-manager-multinode-demo 1/1 Running 0 5m57s
kube-system kube-proxy-kkn5t 1/1 Running 0 5m1s
kube-system kube-proxy-n76jj 1/1 Running 0 5m40s
kube-system kube-scheduler-multinode-demo 1/1 Running 0 5m57s
kube-system storage-provisioner 1/1 Running 2 (5m31s ago) 5m49s
[work@el7 minikube]$
开启kube-proxy的ipvs模式：
#编辑 kube-proxy 配置以启用 ipvs 模式。您可以使用以下命令编辑配置：
kubectl edit configmap kube-proxy -n kube-system
#在配置中，找到 mode 字段并将其设置为 ipvs。
#mode: "ipvs"
#重新启动 kube-proxy 以应用新配置：
kubectl rollout restart daemonset kube-proxy -n kube-system
#验证 ipvs 模式是否已启用。您可以使用以下命令检查 kube-proxy 的日志：
kubectl -n kube-system logs kube-proxy-*
#日志中应该包含以下行：
#Using ipvs Proxier
添加nginx服务：
通过minikube image pull命令下载镜像：
#pull镜像
minikube -p multinode-demo image pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/nginx:1.27.0
#查看镜像id
minikube -p multinode-demo image list --format=table
#打新的镜像tag
minikube -p multinode-demo image tag {image_id} docker.io/nginx:1.27.0或者通过minikube image load命令加载镜像文件：
[work@el7 minikube]$ minikube profile list
|----------------|-----------|---------|--------------|------|---------|---------|-------|----------------|--------------------|
| Profile | VM Driver | Runtime | IP | Port | Version | Status | Nodes | Active Profile | Active Kubecontext |
|----------------|-----------|---------|--------------|------|---------|---------|-------|----------------|--------------------|
| multinode-demo | docker | docker | 192.168.49.2 | 8443 | v1.30.0 | Running | 2 | | * |
|----------------|-----------|---------|--------------|------|---------|---------|-------|----------------|--------------------|[work@el7 minikube]$ minikube image list -p multinode-demo
registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner:v5
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kindnetd:v20240202-8f1494ea
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.12-0
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.11.1[work@el7 minikube]$ minikube image load nginx_1.27.0.tar -p multinode-demo
[work@el7 minikube]$
[work@el7 minikube]$ minikube image list -p multinode-demo
registry.cn-hangzhou.aliyuncs.com/google_containers/storage-provisioner:v5
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.30.0
registry.cn-hangzhou.aliyuncs.com/google_containers/kindnetd:v20240202-8f1494ea
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.12-0
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.11.1
docker.io/library/nginx:1.27.0
[work@el7 minikube]$
Deployment配置：nginx-deployment-2pod.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 100%
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: docker.io/nginx:1.27.0
imagePullPolicy: Never
resources:
requests:
memory: "50Mi"
limits:
memory: "200Mi"
ports:
- name: http
containerPort: 80
terminationGracePeriodSeconds: 6Service配置： nginx-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
type: NodePort
selector:
app: nginx
ports:
- protocol: TCP
nodePort: 30080
port: 8080
targetPort: 80
externalTrafficPolicy: Local部署服务：
kubectl apply -f nginx-deployment-2pod.yaml
kubectl apply -f nginx-svc.yaml开启仪表盘：
查看仪表盘使用的镜像：
minikube addons images dashboard
加载项使用自定义镜像：
因为国内无法访问原站，访问国内代理站sha256又不一致，所以需要更换自定义镜像。
minikube -p multinode-demo addons enable dashboard \
--images='Dashboard=ddn-k8s/docker.io/kubernetesui/dashboard:v2.7.0,MetricsScraper=ddn-k8s/docker.io/kubernetesui/metrics-scraper:v1.0.8' \
--registries='Dashboard=swr.cn-north-4.myhuaweicloud.com,MetricsScraper=swr.cn-north-4.myhuaweicloud.com'执行输出：
metrics-server同样需要自定义镜像：
minikube addons images metrics-server
更换metrics-server镜像：
minikube -p multinode-demo addons enable metrics-server \
--images='MetricsServer=ddn-k8s/registry.k8s.io/metrics-server/metrics-server:v0.7.1' \
--registries='MetricsServer=swr.cn-north-4.myhuaweicloud.com'执行输出：
查看运行pod：
kubectl get pods -A
打开仪表盘url：
minikube -p multinode-demo dashboard --url --port 9091
设置非localhost代理：
kubectl proxy --address='0.0.0.0' --accept-hosts='^*$' -p 9090
宿主机浏览器访问：
------
遇到的错误1，kubelet启动pod失败：
[kubelet-check] Initial timeout of 40s passed.
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration
查看kubelet日志：
sudo tail -f /var/log/messages | grep kubelet如下：
kuberuntime_manager.go:782] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed pulling image \"registry.k8s.io/pause:3.6\": Error response from daemon: Head \"https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.6\": dial tcp 74.125.23.82:443: connect: connection refused" pod="kube-system/kube-controller-manager-control-plane.minikube.internal"
解决办法：
docker pull k8simage/pause:3.6
id=`docker images -q k8simage/pause:3.6`
docker tag $id registry.k8s.io/pause:3.6遇到的错误2，kubectl非root账号执行报错：
执行 kubectl get pod 显示：error: unable to read client-key: permission denied
解决办法：
sudo cp -r /root/.minikube/ $HOME/.minikube/
sudo chown $USER -R $HOME/.minikube/再次执行 kubectl get pod 显示：x509: certificate signed by unknown authority
解决办法：
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config遇到的错误3，node状态NotReady：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
control-plane.minikube.internal NotReady <none> 28m v1.26.1
查看kubelet日志：kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
解决办法：
#安装bridge、vlan、macvlan、tuning等二进制插件
curl -O -L https://github.com/containernetworking/plugins/releases/download/v1.2.0/cni-plugins-linux-amd64-v1.2.0.tgz
mkdir -p /opt/cni/bin
tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-v1.2.0.tgz遇到的错误4，minikube重启报错certificate apiserver not signed：
error execution phase certs/apiserver: [certs] certificate apiserver not signed by CA certificate ca: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "minikubeCA")
解决办法：
#重来一遍
minikube stop
minikube delete
minikube start --image-mirror-country='cn' --kubernetes-version='v1.26.1' --driver=none --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers遇到的错误5，The 'none driver does not support multiple profiles
当试图启动多个node时：
minikube start --image-mirror-country='cn' -v=5 --kubernetes-version='v1.26.1' --driver=none --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --nodes 2 -p multinode-demo报错：X Exiting due to DRV_UNSUPPORTED_PROFILE: The 'none driver does not support multiple profiles: https://minikube.sigs.k8s.io/docs/reference/drivers/none/
错误6：拉取镜像失败
#示例
docker pull gcr.io/k8s-minikube/kicbase:v0.0.44
#改为
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.44
#docker images查询IMAGE ID
docker tag {IMAGE ID} gcr.io/k8s-minikube/kicbase:v0.0.44错误7：permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
解决办法：
sudo usermod -aG docker $USER && newgrp docker错误8：ErrImageNeverPull: Container image "xxxx:vxx" is not present with pull policy of Never
解决办法：
错误9：因 RUNTIME_ENABLE 错误而退出：Failed to start container runtime: Temporary Error: sudo /usr/bin/crictl version
解决办法：
查看docker启动参数：
/usr/bin/dockerd -H unix:///var/run/docker.sock --containerd=/run/containerd/containerd.sock
为minikube设定参数：--cri-socket='/run/containerd/containerd.sock'
--container-runtime=docker --driver=docker --cri-socket='/run/containerd/containerd.sock'
错误10：Failed to connect to bus: No such file or directory
解决办法：
docker@minikube:~$ systemctl restart docker
Failed to connect to bus: No such file or directory
docker@minikube:~$ sudo -s
root@minikube:/home/docker#
root@minikube:/home/docker# systemctl restart docker错误11：node status is NotReady
[work@el7 k8s]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
multinode-demo Ready control-plane 2d20h v1.30.0
multinode-demo-m02 NotReady <none> 2d20h v1.30.0
解决办法：
[work@el7 k8s]$ kubectl describe node multinode-demo-m02
Name: multinode-demo-m02
Roles: <none>
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
Ready Unknown Sat, 03 Aug 2024 00:22:53 +0800 Mon, 05 Aug 2024 01:38:28 +0800 NodeStatusUnknown Kubelet stopped posting node status.
[work@el7 k8s]$ docker exec -it c5f7c9d87677 bash
root@multinode-demo-m02:/# systemctl status kubelet
root@multinode-demo-m02:/# systemctl start kubelet
root@multinode-demo-m02:/# systemctl status kubelet错误12：apt install ipvsadm: E: Unable to locate package ipvsadm
解决办法：
apt update
apt install ipvsadm错误13：cannot open '/var/log/message' for reading: No such file or directory
容器内通过systemd-journald记录的系统日志，日志文件默认存放目录是：
/run/log/journal/
查看办法：
journalctl -n 30 -f错误14：container init: open /proc/sys/net/ipv4/ip_unprivileged_port_start: no such file or directory: unknown
原因：容器里的容器启动了80端口，被限制。
解决办法（要求linux内核版本不低于4.11）：
#临时生效
sysctl net.ipv4.ip_unprivileged_port_start=0
#永久生效
echo "net.ipv4.ip_unprivileged_port_start=0" >> /etc/sysctl.conf错误15：Exiting due to SVC_URL_TIMEOUT: http://127.0.0.1:43480/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ is not accessible: Temporary Error: unexpected response code: 503
解决办法：
#打印更多日志
minikube dashboard --alsologtostderr -p multinode-demo
#查看dashboard的pod状态
kubectl get pods -A
#查看dashboard的pod详情
kubectl describe pod kubernetes-dashboard-779776cb65-kbklx -n kubernetes-dashboard显示错误：
Warning Failed 7m40s (x20 over 173m) kubelet (combined from similar events): Failed to pull image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93": error pulling image configuration: download failed after attempts=6: dial tcp 69.171.247.32:443: i/o timeout
Normal BackOff 5m16s (x436 over 3h36m) kubelet Back-off pulling image "docker.io/kubernetesui/dashboard:v2.7.0@sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93"
更换国内镜像：
#拉取国内镜像
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/kubernetesui/dashboard:v2.7.0
#修改tag
docker tag swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/kubernetesui/dashboard:v2.7.0 docker.io/kubernetesui/dashboard:v2.7.0
#加载到minikube
docker save -o dashboard_v2.7.0.tar kubernetesui/dashboard:v2.7.0
minikube image load dashboard_v2.7.0.tar -p multinode-demo
#删除旧pod重新构建
kubectl delete pod kubernetes-dashboard-779776cb65-kbklx -n kubernetes-dashboard
minikube dashboard --alsologtostderr -p multinode-demo终极大法从头再来：
rm -drf ~/.minikube/多节点支持相关文档：
https://minikube.kubernetes.ac.cn/docs/tutorials/multi_node/
https://minikube.sigs.k8s.io/docs/drivers/none/
Using Multi-Node Clusters | minikube
Add `local-path-provisioner` addon by presztak · Pull Request #15062 · kubernetes/minikube · GitHub
扫一扫
332
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
