Kubernetes/KubeSphere 安装踩坑记:从 context deadline exceeded 到成功部署的完整排障笔记

目录

Kubernetes/KubeSphere 安装踩坑记:从 context deadline exceeded 到成功部署的完整排障笔记

一、问题现象

二、第一手日志采集

三、定位思路

四、分步解决

4-1 处理 pause:3.8

4-2 处理 kube-apiserver:v1.31.0

五、再次安装并验证

六、经验总结

七、尾声


Kubernetes/KubeSphere 安装踩坑记:从 context deadline exceeded 到成功部署的完整排障笔记

适用版本:Kubernetes 1.31.x / KubeSphere 4.1
运行时:containerd
操作系统:CentOS 7.9


一、问题现象

执行 ./kk create cluster -f config-sample.yaml 初始化节点,安装脚本在 wait-control-plane 阶段反复重试并最终报错:

context deadline exceeded
error execution phase wait-control-plane: could not initialize a Kubernetes cluster

此时主节点 kubelet 日志刷屏,集群始终无法启动。


二、第一手日志采集

# kubelet 当前状态
systemctl status kubelet -l

# 10 分钟内关键错误
sudo journalctl -u kubelet --since "10 minutes ago" | grep -Ei "error|failed|cgroup"

核心错误片段:

failed to get sandbox image "registry.k8s.io/pause:3.8":
dial tcp 173.194.203.82:443: i/o timeout

以及后续安装 KubeSphere 组件时:

failed to pull image "docker.io/kubesphere/kube-apiserver:v1.31.0":
500 Internal Server Error

三、定位思路

线索结论
pod sandbox 创建失败 无法拉取 pause:3.8根因:国外镜像仓库被墙,pull 超时
kube-apiserver 镜像 500 错误Docker Hub 临时故障 / 国内网络劣化

两个核心镜像拉取失败 → kubelet 无法启动沙箱容器 → control plane 卡死。


四、分步解决

详细离线导入方法请见:
《从零实现 registry.k8s.io/pause:3.8 镜像的导出与导入》
《从零实现 crictl pull kubesphere/kube-apiserver:v1.31.0 的全流程操作》

4-1 处理 pause:3.8
  1. 在能出网的跳板机

    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8
    docker tag  registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8 \
                registry.k8s.io/pause:3.8
    docker save registry.k8s.io/pause:3.8 -o pause-3.8.tar
    
  2. 拷贝至目标节点并导入 containerd

    scp pause-3.8.tar node1:/tmp/
    sudo ctr -n k8s.io images import /tmp/pause-3.8.tar
    
  3. 确认导入成功

    crictl images | grep pause
    
4-2 处理 kube-apiserver:v1.31.0
  1. 预拉镜像并导出

    docker pull kubesphere/kube-apiserver:v1.31.0
    docker save kubesphere/kube-apiserver:v1.31.0 -o kube-apiserver-1.31.0.tar
    
  2. 目标节点导入

    sudo ctr -n k8s.io images import kube-apiserver-1.31.0.tar
    

技巧:直接用 crictl pull --image-endpoint 也行,但离线文件更稳妥,可反复使用。


五、再次安装并验证

./kk create cluster -f config-sample.yaml --with-local-storage
  • kubelet 不再报 failed to get sandbox image

  • control-plane 组件全部 Running

  • kubectl get node -o wide 显示所有节点 Ready


六、经验总结

类别建议
镜像策略对所有 基础镜像(pause、coredns、etcd)控制面镜像 预先离线导入,彻底规避外网依赖
containerd 命名空间使用 ctr -n k8s.io images import,否则镜像导入到 default 后 kubelet 依旧找不到
crictl 调试`crictl ps -a
自动化脚本建议编写 image-cache.sh 批量处理 docker pull/tag/save,上线时一键导入
网络排障镜像 HEAD/GET 超时 ≈ DNS 劫持 / 443 被阻断,优先切换国内镜像源或自建 Harbor

七、尾声

这次“context deadline exceeded” 看似是 kubeadm 的错误,实则暴露了离线镜像策略的不完备。只要掌握 日志定位 → 镜像补全 → 再次验证 的闭环套路,类似问题能在 10 分钟内搞定。希望本文的实战笔记能帮助你在下一次部署 Kubernetes / KubeSphere 时一路绿灯 🚦。

以下是完整的错误记录

报错详情

Unfortunately, an error has occurred:
        context deadline exceeded

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
        - 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
        Once you have found the failing container, you can inspect its logs with:
        - 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
error execution phase wait-control-plane: could not initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher: Process exited with status 1
14:35:44 CST retry: [node1]

检查日志 systemctl status kubelet

看到报错内容

Apr 23 14:39:42 node1 kubelet[44232]: E0423 14:39:42.697307   44232 eviction_manager.go:285] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"node1\" not found"
Apr 23 14:39:48 node1 kubelet[44232]: E0423 14:39:48.736613   44232 event.go:368] "Unable to write event (may retry after sleeping)" err="Post \"https://lb.kubesphere.local:6443/api/v1/namespaces/default/events\": dial tcp 10.33.34.166:6443: connect: connection refused" event="&Event{ObjectMeta:{node1.1838df18aab26a9c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:node1,UID:node1,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node node1 status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:node1,},FirstTimestamp:2025-04-23 14:35:52.642095772 +0800 CST m=+0.729483809,LastTimestamp:2025-04-23 14:35:52.642095772 +0800 CST m=+0.729483809,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:node1,}"
Apr 23 14:39:49 node1 kubelet[44232]: E0423 14:39:49.241750   44232 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://lb.kubesphere.local:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/node1?timeout=10s\": dial tcp 10.33.34.166:6443: connect: connection refused" interval="7s"
Apr 23 14:39:49 node1 kubelet[44232]: I0423 14:39:49.468021   44232 kubelet_node_status.go:72] "Attempting to register node" node="node1"
Apr 23 14:39:49 node1 kubelet[44232]: E0423 14:39:49.468592   44232 kubelet_node_status.go:95] "Unable to register node with API server" err="Post \"https://lb.kubesphere.local:6443/api/v1/nodes\": dial tcp 10.33.34.166:6443: connect: connection refused" node="node1"
Apr 23 14:39:52 node1 kubelet[44232]: E0423 14:39:52.698356   44232 eviction_manager.go:285] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"node1\" not found"
Apr 23 14:39:53 node1 systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
Apr 23 14:39:53 node1 systemd[1]: kubelet.service: Deactivated successfully.
Apr 23 14:39:53 node1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Apr 23 14:39:53 node1 systemd[1]: kubelet.service: Consumed 5.322s CPU time.

仔细检查日志, 发现报错

sudo journalctl -u kubelet --since "10 minutes ago" | grep -iE "error|cgroup|failed"

Apr 23 15:17:13 node1 kubelet[48293]: E0423 15:17:13.238787   48293 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8\": dial tcp 173.194.203.82:443: i/o timeout"

解决pause:3.8的问题参考文章 从零实现 registry.k8s.io/pause:3.8 镜像的导出与导入

另外查看日志也可能遇到 kube-apiserver下载失败的情况

pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull kubesphere/kube-apiserver:v1.31.0" 
E0424 10:32:36.669380   14293 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"docker.io/kubesphere/kube-apiserver:v1.31.0\": failed to resolve reference \"docker.io/kubesphere/kube-apiserver:v1.31.0\": unexpected status from HEAD request to https://docker.1ms.run/v2/kubesphere/kube-apiserver/manifests/v1.31.0?ns=docker.io: 500 Internal Server Error" image="kubesphere/kube-apiserver:v1.31.0"
FATA[0031] pulling image: failed to pull and unpack image "docker.io/kubesphere/kube-apiserver:v1.31.0": failed to resolve reference "docker.io/kubesphere/kube-apiserver:v1.31.0": unexpected status from HEAD request to https://docker.1ms.run/v2/kubesphere/kube-apiserver/manifests/v1.31.0?ns=docker.io: 500 Internal Server Error: Process exited with status 1
10:32:36 CST retry: [node1]

参考: 从零实现 crictl pull kubesphere/kube-apiserver:v1.31.0 的全流程操作

然后再执行安装即可成功了

./kk create cluster -f config-sample.yaml --with-local-storage

### 解决方案 当遇到 `Post http://ipc/engine/switch context deadline exceeded` 错误时,这通常意味着客户端等待服务器响应的时间超过了设定的超时时间。对于 Windows 引擎而言,RPC 调用失败可能是由于多种原因引起的。 #### 1. 配置调整 增加请求超时设置可以有效缓解此类问题的发生。通过修改应用程序配置文件中的超时参数来延长 RPC 请求的最大允许等待时间[^1]: ```yaml rpc: timeout: 60s # 将默认值改为更长的时间间隔 ``` #### 2. 网络连接验证 确认主机与目标机器之间的网络连通性良好,并确保防火墙或其他安全软件未阻止必要的端口通信。测试命令如下所示: ```powershell Test-NetConnection -ComputerName target_machine_ip -Port rpc_port_number ``` #### 3. 服务状态检查 确保远程过程调用 (RPC) 和其他依赖的服务正在运行并处于活动状态。可以通过 PowerShell 来查看和启动这些服务: ```powershell Get-Service RpcSs, DcomLaunch | Where-Object {$_.Status -ne 'Running'} | Start-Service ``` #### 4. 日志分析 收集详细的日志信息有助于进一步诊断具体的原因所在。启用调试级别的日志录功能可以帮助定位潜在的问题根源: ```xml <configuration> <system.diagnostics> <sources> <source name="System.ServiceModel.MessageLogging"> <listeners> <add type="System.Diagnostics.XmlWriterTraceListener" initializeData="C:\Logs\Traces.svclog"/> </listeners> </source> </sources> </system.diagnostics> </configuration> ``` #### 5. 版本兼容性审查 最后但同样重要的是,核查所使用的库版本是否完全支持当前的操作系统环境以及是否存在已知 bug 或者不兼容的情况。如果适用的话,请尝试更新至最新稳定版以获得更好的稳定性和支持.
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值