k8s中pod启动报存储相关问题总结

报错1: Volume已经被绑定

Multi-Attach error for Volume is already exclusively attached to one node and can't be attached to another

Warning FailedMount 44s (x2 over 108s) kubelet MountVolume.MountDevice failed for volume "pvc-e09f23da-f21b-4365-a24b-528d026355ee" : rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0009-rook-ceph-000000000000000a-32b04bb7-ae3e-11eb-a9c8-ae53b05a732a already exists

原因

• pod不是被正常关闭，强制删除有可能没有删除pod与volume绑定关系

如何解决

# 获取绑定关系
kubectl -n newpower get volumeattachment
# 删除pod所关联的volumeattachment
kubectl -n newpower delete volumeattachment csi-fcfe0c84e304b8a452669e1015684edfd74914c67efe62f214cd9ce71090a51d

• 还是无法解决，删除(重启)rook-ceph下的rdb服务
  • csi-rbdplugin
  • csi-rbdplugin-provisioner

报错2: rbd image XXX is still being used

原因

• pod非正常关闭

如何解决

• 进入rook-ceph的工具util中，执行

查看绑定的ip地址
rbd status replica-cms-pool/csi-vol-480c78d4-330f-11ed-a423-7abfc702790e
示例输出：
Watchers:
	watcher=10.244.0.0:0/2040186430 client.25344332 cookie=18446462598732840968
将该ip加入黑名单
ceph osd blocklist add 10.244.0.0:0/2040186430