记录k8s glusterfs pod 存储卷挂载异常

背景：发现大量的 pod 处于异常状态，通过 describe 这些pod 发现都报类似如下的挂载异常

error while creating mount source path '/var/lib/kubelet/pods/507663c2-addf-4cb6-82d9-c9bfdb6f1ba1/volumes/kubernetes.io~glusterfs/pvc-5cb50d58-8564-4851-8823-d4676744aa7d': mkdir /var/lib/kubelet/pods/507663c2-addf-4cb6-82d9-c9bfdb6f1ba1/volumes/kubernetes.io~glusterfs/pvc-5cb50d58-8564-4851-8823-d4676744aa7d: file exists

发现 glusterfs k8s 存储后端某个 pod 重启过、

重启各个服务正常启动

原因待探究

衍生问题

需要prometheus pod 挂载存储卷，但是发现 pod 一直处于pending 状态，进而查看pvc 处于pending 状态 Failed to provision volume with StorageClass "gluster-heketi": failed to create volume: failed to create volume: see kube-controller-manager.log for details

查看 heketi pod 日志，报连接问题 transport endpoint is not connected

[negroni] 2022-12-29T10:47:31+08:00 | 500 |      4.234284ms | 10.1.71.95:8080 | POST /volumes
[heketi] ERROR 2022/12/29 10:47:31 heketi/apps/glusterfs/operations_manage.go:220:glusterfs.AsyncHttpOperation: Create Volume Build Failed: write /var/lib/heketi/heketi.db: transport endpoint is not connected

因为之前在k8s mysql 碰到过，尝试着删除 heketi pod，待其自动重部署后，pvc 成功分配绑定

挂载异常：Read-only file system

删除glusterfs pod ，待其重启