kubernetes(k8s)异常处理积累-持续更新

全栈行动派

已于 2023-02-11 11:54:01 修改

阅读量1.9k

点赞数

分类专栏： kubernetes 文章标签： kubernetes docker 容器云原生 Powered by 金山文档

于 2023-02-06 00:30:00 首次发布

本文链接：https://blog.csdn.net/u011837804/article/details/128693933

版权

kubernetes 专栏收录该内容

45 篇文章 42 订阅

订阅专栏

1、k8s1.25创建StorageClass后使用时，PVC状态一直Pending状态

1.1、异常集群信息

[root@k8s-master nfs]# kubectl cluster-info
Kubernetes control plane is running at https://10.211.55.11:6443
CoreDNS is running at https://10.211.55.11:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@k8s-master nfs]#
[root@k8s-master nfs]# kubectl get nodes -owide
NAME         STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE          KERNEL-VERSION          CONTAINER-RUNTIME
k8s-master   Ready    control-plane   21h   v1.25.0   10.211.55.11   <none>        CentOS Stream 8   4.18.0-408.el8.x86_64   docker://20.10.22
k8s-node1    Ready    <none>          21h   v1.25.0   10.211.55.12   <none>        CentOS Stream 8   4.18.0-408.el8.x86_64   docker://20.10.22
k8s-node2    Ready    <none>          21h   v1.25.0   10.211.55.13   <none>        CentOS Stream 8   4.18.0-408.el8.x86_64   docker://20.10.22
[root@k8s-master nfs]#

1.2、NFS-Subdir-External-Provisioner清单文件信息

kind: Deployment
apiVersion: apps/v1
metadata:
  name: nfs-client-provisioner
  namespace: dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nfs-client-provisioner
  strategy:
    type: Recreate        ## 设置升级策略为删除再创建(默认为滚动更新)
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      containers:
        - name: nfs-client-provisioner
          image: registry.cn-beijing.aliyuncs.com/mydlq/nfs-subdir-external-provisioner:v4.0.0
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME  # Provisioner的名称,以后设置的storageclass要和这个保持一致
              value: storage-nfs
            - name: NFS_SERVER        # NFS服务器地址,需和valumes参数中配置的保持一致
              value: 10.211.55.11
            - name: NFS_PATH          # NFS服务器数据存储目录,需和valumes参数中配置的保持一致
              value: /root/data/nfs
            - name: ENABLE_LEADER_ELECTION
              value: "true"
      volumes:
        - name: nfs-client-root
          nfs:
            server: 10.211.55.11        # NFS服务器地址
            path: /root/data/nfs        # NFS共享目录

1.3、StorageClass清单文件信息

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  namespace: dev
  name: nfs-storage
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"  ## 是否设置为默认的storageclass
provisioner: storage-nfs                                   ## 动态卷分配者名称，必须和上面创建的deploy中环境变量“PROVISIONER_NAME”变量值一致
parameters:
  archiveOnDelete: "true"                                 ## 设置为"false"时删除PVC不会保留数据,"true"则保留数据
mountOptions: 
  - hard                                                  ## 指定为硬挂载方式
  - nfsvers=2                                             ## 指定NFS版本,这个需要根据NFS Server版本号设置

1.4、pvc清单文件

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: storage-pvc
  namespace: dev
spec:
  storageClassName: nfs-storage    ## 需要与上面创建的storageclass的名称一致
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Mi

1.5、pvc异常信息

[root@k8s-master nfs]# kubectl get pvc -n dev
NAME          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
storage-pvc   Pending                                      nfs-storage    20m
[root@k8s-master nfs]#
[root@k8s-master nfs]#
[root@k8s-master nfs]#
[root@k8s-master nfs]# kubectl describe pvc -n dev storage-pvc
Name:          storage-pvc
Namespace:     dev
StorageClass:  nfs-storage
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: storage-nfs
               volume.kubernetes.io/storage-provisioner: storage-nfs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age                 From                         Message
  ----    ------                ----                ----                         -------
  Normal  ExternalProvisioning  40s (x83 over 20m)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "storage-nfs" or manually created by system administrator

1.6、pod日志信息

[root@k8s-master nfs]# kubectl get pod -n dev
NAME                                     READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-59b496764-5kts2   1/1     Running   0          22m
[root@k8s-master nfs]#
[root@k8s-master nfs]#
[root@k8s-master nfs]# kubectl logs nfs-client-provisioner-59b496764-5kts2 -n dev
I0115 04:27:18.336963       1 leaderelection.go:242] attempting to acquire leader lease  dev/storage-nfs...
E0115 04:27:18.338810       1 leaderelection.go:320] error retrieving resource lock dev/storage-nfs: endpoints "storage-nfs" is forbidden: User "system:serviceaccount:dev:default" cannot get resource "endpoints" in API group "" in the namespace "dev"

1.7、异常解读

error retrieving resource lock dev/storage-nfs: endpoints "storage-nfs" is forbidden: User "system:serviceaccount:dev:default" cannot get resource "endpoints" in API group "" in the namespace "dev"

这个异常信息意思是用户 "system:serviceaccount:dev:default" 没有权限访问K8s的 API group的。

此次异常我们用的namespace为dev，在我们创建namespace时会默认为namespace分配一个serviceAccount，可以通过 “kubectl get sa -n 你的命名空间” 获取 namespace绑定的serviceAccount

[root@k8s-master ~]# kubectl get sa -n dev
NAME      SECRETS   AGE
default   0         4h12m

1.8、异常解决

知道了报错的原因，我们就可以解决这个问题了，那就是创建一个新的ServiceAccount绑定到我们的pod上。

而新创建的ServiceAccount需要有 API group的权限。说到权限有涉及到新的知识点，就是K8s的授权插件，我这里用的是RBAC，意思就是根据角色来控制权限。

ServiceAccount对象代表一个账号，则我们还需要一个role对象和一个role与ServiceAccount绑定的rolebinding对象，这些都是RBAC插件提供的资源对象

如果还是不会，可以参考这篇文章，创建ServiceAccount https://blog.csdn.net/u011837804/article/details/128692744

2、k8s1.25动态卷挂载失败

Warning FailedScheduling 3m39s default-scheduler 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
Normal Scheduled 3m38s default-scheduler Successfully assigned dev/redis-cm-0 to k8s-node1
Warning FailedMount 87s kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data logs timezone kube-api-access-s8b2z conf]: timed out waiting for the condition
Warning FailedMount 54s (x9 over 3m26s) kubelet MountVolume.SetUp failed for volume "pvc-1970c511-f218-44ef-bb23-673a5fd4f12d" : mount failed: exit status 32
Mounting command（挂载命令）: mount
Mounting arguments（挂载参数）: -t nfs -o hard,nfsvers=2 10.211.55.11:/root/data/nfs/dev-data-redis-cm-0-pvc-1970c511-f218-44ef-bb23-673a5fd4f12d /var/lib/kubelet/pods/921ae61e-8d0d-4d68-9fff-7b48d780e6c9/volumes/kubernetes.io~nfs/pvc-1970c511-f218-44ef-bb23-673a5fd4f12d
Output: mount.nfs: requested NFS version or transport protocol is not supported

注意：错误信息主要说明为 Output: mount.nfs: requested NFS version or transport protocol is not supported（版本号或者协议不支持）

知道问题原因了，那就好说了，要么服务端和客户端版本对不上，要么协议对不上

在挂载参数中有个很重要的参数

服务端和客户端nfs版本信息 “nfsvers=2”意思是使用那个版本的协议

# 服务端版本号
[root@k8s-master ~]# nfsstat -v
Server packet stats:
packets    udp        tcp        tcpconn
701        0          701        195

Server rpc stats:
calls      badcalls   badfmt     badauth    badclnt
515        186        186        0          0

Server reply cache:
hits       misses     nocache
0          0          515

Server io stats:
read       write
0          0

Server read ahead cache:
size       0-10%      10-20%     20-30%     30-40%     40-50%     50-60%     60-70%     70-80%     80-90%     90-100%    notfound
32         0          0          0          0          0          0          0          0          0          0          0

Server file handle cache:
lookup     anon       ncachedir  ncachenondir  stale
0          0          0          0          0

Server nfs v4:
null             compound
8         1%     507      98%

Server nfs v4 operations:
op0-unused       op1-unused       op2-future       access           close
0         0%     0         0%     0         0%     31        2%     0         0%
commit           create           delegpurge       delegreturn      getattr
0         0%     5         0%     0         0%     0         0%     294      23%
getfh            link             lock             lockt            locku
46        3%     0         0%     0         0%     0         0%     0         0%
lookup           lookup_root      nverify          open             openattr
43        3%     0         0%     0         0%     0         0%     0         0%
open_conf        open_dgrd        putfh            putpubfh         putrootfh
0         0%     0         0%     304      23%     0         0%     17        1%
read             readdir          readlink         remove           rename
0         0%     3         0%     0         0%     0         0%     3         0%
renew            restorefh        savefh           secinfo          setattr
0         0%     0         0%     3         0%     0         0%     5         0%
setcltid         setcltidconf     verify           write            rellockowner
0         0%     0         0%     0         0%     0         0%     0         0%
bc_ctl           bind_conn        exchange_id      create_ses       destroy_ses
0         0%     0         0%     17        1%     10        0%     8         0%
free_stateid     getdirdeleg      getdevinfo       getdevlist       layoutcommit
0         0%     0         0%     0         0%     0         0%     0         0%
layoutget        layoutreturn     secinfononam     sequence         set_ssv
0         0%     0         0%     8         0%     465      36%     0         0%
test_stateid     want_deleg       destroy_clid     reclaim_comp     allocate
0         0%     0         0%     7         0%     9         0%     0         0%
copy             copy_notify      deallocate       ioadvise         layouterror
0         0%     0         0%     0         0%     0         0%     0         0%
layoutstats      offloadcancel    offloadstatus    readplus         seek
0         0%     0         0%     0         0%     0         0%     0         0%
write_same
0         0%

[root@k8s-master ~]#

# 客户端版本号
[root@k8s-node1 kubernetes.io~nfs]# nfsstat -c
Client rpc stats:
calls      retrans    authrefrsh
515        0          515

Client nfs v4:
null             read             write            commit           open
8         1%     0         0%     0         0%     0         0%     0         0%
open_conf        open_noat        open_dgrd        close            setattr
0         0%     0         0%     0         0%     0         0%     5         0%
fsinfo           renew            setclntid        confirm          lock
24        4%     0         0%     0         0%     0         0%     0         0%
lockt            locku            access           getattr          lookup
0         0%     0         0%     31        6%     40        7%     43        8%
lookup_root      remove           rename           link             symlink
8         1%     0         0%     3         0%     0         0%     0         0%
create           pathconf         statfs           readlink         readdir
5         0%     16        3%     91       17%     0         0%     3         0%
server_caps      delegreturn      getacl           setacl           fs_locations
40        7%     0         0%     0         0%     0         0%     0         0%
rel_lkowner      secinfo          fsid_present     exchange_id      create_session
0         0%     0         0%     0         0%     17        3%     10        1%
destroy_session  sequence         get_lease_time   reclaim_comp     layoutget
8         1%     138      26%     1         0%     9         1%     0         0%
getdevinfo       layoutcommit     layoutreturn     secinfo_no       test_stateid
0         0%     0         0%     0         0%     8         1%     0         0%
free_stateid     getdevicelist    bind_conn_to_ses destroy_clientid seek
0         0%     0         0%     0         0%     7         1%     0         0%
allocate         deallocate       layoutstats      clone
0         0%     0         0%     0         0%     0         0%

通过检查服务端和客户端nfs版本发现，版本都是4，那就看协议是否能对上了

在挂载参数中使用的2版本的协议，版本对不上，将2改为4即可

如果不会改的朋友，请参考此篇文章https://mp.csdn.net/mp_blog/creation/editor/new/128692744