1. 环境
docker版本:
Client:
Version: 1.13.1
API version: 1.26
Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
centos信息:
Linux host-centos7.0-temp 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
2. 问题
停止容器后,无法重启,报错原因:大致是目录被占用Device or resource busy
容易状态dead以及exit
Error response from daemon: Cannot start container XXX: Error getting container ddf1dd91bbf46dc648268327f8f7c6fffaf2f19cda5cf1d97fdc701016d4332c from driver devicemapper: Error mounting 目录': device or resource busy
删除、卸载目录均报错Device or resource busy:
rm -rf /docker-root/devicemapper/mnt/7d7200e04c5a166606b5bade185af33b7a9e85c9a0f49a3331eb58a0fdc4b4ae
umount /docker-root/devicemapper/mnt/7d7200e04c5a166606b5bade185af33b7a9e85c9a0f49a3331eb58a0fdc4b4ae
3. 解决办法
存在mount泄漏,编写脚本获取相应占用目录进程:
[root@host-centos7 deploy]# cat leak_rbd.sh
#!/bin/bash
declare -A map
for i in `find /proc/*/mounts -exec grep $1 {} + 2>/dev/null | awk '{print $1"#"$2}'`
do
pid=`echo $i | awk -F "[/]" '{print $3}'`
point=`echo $i | awk -F "[#]" '{print $2}'`
mnt=`ls -l /proc/$pid/ns/mnt |awk '{print $11}'`
map["$mnt"]="exist"
cmd=`cat /proc/$pid/cmdline`
echo -e "$pid\t$mnt\t$cmd\t$point"
done
for i in `ps aux|grep docker-containerd-shim |grep -v "grep" |awk '{print $2}'`
do
mnt=`ls -l /proc/$i/ns/mnt 2>/dev/null | awk '{print $11}'`
if [[ "${map[$mnt]}" == "exist" ]];then
ps aux |grep $i |grep -v "grep" |awk '{print $11" "$12}'
fi
done
执行:
[root@host-centos7 deploy]# sh leak_rbd.sh /hel/docker-root/devicemapper/mnt/7d7200e04c5a166606b5bade185af33b7a9e85c9a0f49a3331eb58a0fdc4b4ae
**4647** mnt:[4026536961] /usr/sbin/ntpd-untp:ntp-g /hel/docker-root/devicemapper/mnt/7d7200e04c5a166606b5bade185af33b7a9e85c9a0f49a3331eb58a0fdc4b4ae
[root@host-centos7 deploy]# ps -ef |grep 4647
ntp **4647** 1 0 Aug08 ? 00:00:00 /usr/sbin/ntpd -u ntp:ntp -g
root 24886 11976 0 15:35 pts/1 00:00:00 grep --color=auto 4647
如上,发现是ntp服务占用目录,先停止该服务然后删除目录即可。
kill 4647
rm -rf /hel/docker-root/devicemapper/mnt/7d7200e04c5a166606b5bade185af33b7a9e85c9a0f49a3331eb58a0fdc4b4ae
启动docker服务正常!
4. 总结
有说是docker版本存储的bug,其他解决办法如网上说的docker.service中设置MountFlags,没有试验~