K8S故障处理：ImagePullBackOff 错误处理

问题描述

k8s创建pod并运行时：

[root@master ~]# kubectl run nginx --image=nginx
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx created

查看pods：出现下面报错，一直无法执行成功。

[root@master ~]# kubectl get pods
NAME                    READY   STATUS             RESTARTS   AGE
nginx-dbddb74b8-h4mwk   0/1     ImagePullBackOff   0          2m56s

调查错误原因，在 master 服务器上面执行命令: kubectl describe pod ，查看nginx实例的报错信息：

[root@master ~]# kubectl describe pod nginx-dbddb74b8-597r8
Name:               nginx-dbddb74b8-597r8
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               14.0.0.55/14.0.0.55
Start Time:         Thu, 08 Oct 2020 01:19:08 +0800
Labels:             pod-template-hash=dbddb74b8
                    run=nginx
Annotations:        <none>
Status:             Pending
IP:                 172.17.41.3
Controlled By:      ReplicaSet/nginx-dbddb74b8
Containers:
  nginx:
    Container ID:   
    Image:          nginx
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-x5xpc (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-x5xpc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-x5xpc
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age               From                Message
  ----     ------     ----              ----                -------
  Normal   Scheduled  93s               default-scheduler   Successfully assigned default/nginx-dbddb74b8-597r8 to 14.0.0.55
  Warning  Failed     19s               kubelet, 14.0.0.55  Failed to pull image "nginx": rpc error: code = Canceled desc = context canceled
  Warning  Failed     19s               kubelet, 14.0.0.55  Error: ErrImagePull
  Normal   BackOff    18s               kubelet, 14.0.0.55  Back-off pulling image "nginx"
  Warning  Failed     18s               kubelet, 14.0.0.55  Error: ImagePullBackOff
  Normal   Pulling    5s (x2 over 92s)  kubelet, 14.0.0.55  pulling image "nginx"

可以看到报错信息：

 Warning  Failed     19s               kubelet, 14.0.0.55  Failed to pull image "nginx": rpc error: code = Canceled desc = context canceled

原因是服务器忘记了给docker配置国内镜像：

[root@node01 cfg]#  tee /etc/docker/daemon.json <<-'EOF'
>     {
>     "registry-mirrors": ["https://8i185852.mirror.aliyuncs.com"]
>     }
> EOF

配置上国内镜像，然后执行：

[root@node01 cfg]# systemctl daemon-reload
[root@node01 cfg]# systemctl restart docker

就可以了：

[root@master ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   

[root@master ~]# kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-t45hv   1/1     Running   0          34s

搞定。
（或者你直接
systemctl daemon-reload
systemctl restart docker）