测试StatefulSet资源时，pod卡死在ContainerCreating

背景：

创建一个名为myapp的StatefulSet资源，通过Pod模板创建了3个Pod资源副本，并基于volumeClaimTemplates向前面创建的PV进行了请求大小为2Gi的专用存储卷。

---

问题描述：

没有像预计的那样创建三个pod副本，只创建了一个myapp-0副本，并且卡死在ContainerCreating状态。

NAME                           READY   STATUS              RESTARTS   AGE
myapp-0                        0/1     ContainerCreating   0          2m57s

---

原因分析：

查看pv和pvc的情况如下，发现只有一个pv和pvc进行了绑定，并没有像预计的那样有三对绑定的pv和pvc。

[root@k8sMaster volumes]# kubectl get pv
NAME    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS   REASON   AGE
pv001   1Gi        RWO,RWX        Retain           Available                                                       14m
pv002   2Gi        RWO            Retain           Bound       default/myappdata-myapp-0                           14m
pv003   2Gi        RWO,RWX        Retain           Available                                                       14m
pv004   2Gi        RWO,RWX        Retain           Available                                                       14m
pv005   2Gi        RWO,RWX        Retain           Available                                                       14m

[root@k8sMaster volumes]# kubectl get pvc
NAME                STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
myappdata-myapp-0   Bound    pv002    2Gi        RWO                           13m

---

解决方案：

通过kubectl describe pod myapp-0查看此pod的情况，如下：

[root@k8sMaster volumes]# kubectl describe pod myapp-0
Name:           myapp-0
Namespace:      default
Priority:       0
Node:           k8s-node2/192.168.139.145
Start Time:     Tue, 16 Feb 2021 21:10:44 +0800
Labels:         app=myapp-pod
                controller-revision-hash=myapp-8598dd746f
                statefulset.kubernetes.io/pod-name=myapp-0
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  StatefulSet/myapp
Containers:
  myapp:
    Container ID:   
    Image:          ikubernetes/myapp:v1
    Image ID:       
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/share/nginx/html from myappdata (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dcwl2 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  myappdata:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  myappdata-myapp-0
    ReadOnly:   false
  default-token-dcwl2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dcwl2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age        From                Message
  ----     ------       ----       ----                -------
  Normal   Scheduled    <unknown>  default-scheduler   Successfully assigned default/myapp-0 to k8s-node2
  Warning  FailedMount  12m        kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107550.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  12m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107554.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  11m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107577.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  11m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107625.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  11m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107642.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  11m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107673.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  11m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107771.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  10m  kubelet, k8s-node2  MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-107919.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  9m58s                 kubelet, k8s-node2  Unable to attach or mount volumes: unmounted volumes=[myappdata], unattached volumes=[default-token-dcwl2 myappdata]: timed out waiting for the condition
  Warning  FailedMount  104s (x6 over 9m53s)  kubelet, k8s-node2  (combined from similar events): MountVolume.SetUp failed for volume "pv002" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002 --scope -- mount -t nfs store:/data/volumes/v2 /var/lib/kubelet/pods/e944e008-5137-40ac-b719-308810075492/volumes/kubernetes.io~nfs/pv002
Output: Running scope as unit run-110486.scope.
mount.nfs: Failed to resolve server store: Name or service not known
mount.nfs: Operation already in progress
  Warning  FailedMount  52s (x3 over 5m24s)  kubelet, k8s-node2  Unable to attach or mount volumes: unmounted volumes=[myappdata], unattached volumes=[myappdata default-token-dcwl2]: timed out waiting for the condition

发现问题描述在于mount.nfs: Failed to resolve server store: Name or service not known mount.nfs: Operation already in progress Warning FailedMount 52s (x3 over 5m24s) kubelet, k8s-node2 Unable to attach or mount volumes: unmounted volumes=[myappdata], unattached volumes=[myappdata default-token-dcwl2]: timed out waiting for the condition，大概意思是k8s-node2节点无法识别store（部署了nfs的主机名），因此无法挂载nfs卷。

查看k8s-node2节点的/etc/hosts配置，发现忘记配置了store的ip映射关系，故加上store 192.168.139.189，然后重新部署StatefulSet资源成功。
结果如下：

[root@k8sMaster statefulSetDemo]# kubectl get pod -owide
NAME                           READY   STATUS    RESTARTS   AGE    IP            NODE        NOMINATED NODE   READINESS GATES
myapp-0                        1/1     Running   0          11s    10.244.1.27   k8s-node2   <none>           <none>
myapp-1                        1/1     Running   0          10s    10.244.0.17   k8s-node1   <none>           <none>
myapp-2                        1/1     Running   0          5s     10.244.1.28   k8s-node2   <none>           <none>
mydemo01pod-694474c9bd-5pwlc   1/1     Running   0          2d6h   10.244.0.4    k8s-node1   <none>           <none>
mydemo02pod-7cb5c8c875-vh9gv   1/1     Running   0          34h    10.244.0.9    k8s-node1   <none>           <none>

[root@k8sMaster statefulSetDemo]# kubectl get pv,pvc
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS   REASON   AGE
persistentvolume/pv001   1Gi        RWO,RWX        Retain           Available                                                       27m
persistentvolume/pv002   2Gi        RWO            Retain           Bound       default/myappdata-myapp-0                           27m
persistentvolume/pv003   2Gi        RWO,RWX        Retain           Bound       default/myappdata-myapp-1                           27m
persistentvolume/pv004   2Gi        RWO,RWX        Retain           Bound       default/myappdata-myapp-2                           27m
persistentvolume/pv005   2Gi        RWO,RWX        Retain           Available                                                       27m

NAME                                      STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/myappdata-myapp-0   Bound    pv002    2Gi        RWO                           26m
persistentvolumeclaim/myappdata-myapp-1   Bound    pv003    2Gi        RWO,RWX                       26s
persistentvolumeclaim/myappdata-myapp-2   Bound    pv004    2Gi        RWO,RWX                       21s

---

If you miss the train I’m on

You will know that l am gone

You can hear the whistle blow a hundred miles…