当节点出现pod挂载不上,且查看节点日志(/var/log/message)发现不断有“Orphaned pod”字段的日志打印出来,如下:
Jan 21 03:07:08 abc_k8s_worker01 kubelet: E0121 03:07:08.894211 6507 kubelet_volumes.go:140] Orphaned pod "0401c09a-1b0c-11e9-94f8-00163e14dc48" found, but volume paths are still present on disk : There were a total of 4 errors similar to this. Turn up verbosity to see them.
说明当前有pod(0401c09a-1b0c-11e9-94f8-00163e14dc48)为僵尸pod,且在试图删除pod残留数据的时候失败。这时可以在该节点上运行下面脚本解决;
curl -o diagnose_orphanpod.sh http://aliacs-k8s-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/public/diagnose/diagnose_orphanpod.sh
该问题为kubelet处理数据卷时候导致的问题,目前社区正在讨论解决方案:
https://github.com/kubernetes/kubernetes/issues/60987