docker容器退出无法重启Device or resource busy且无法umount

1. 环境

docker版本：

Client:
 Version:         1.13.1
 API version:     1.26
Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)

centos信息：

Linux host-centos7.0-temp 3.10.0-123.el7.x86_64 #1 SMP Mon Jun 30 12:09:22 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

2. 问题

停止容器后，无法重启，报错原因：大致是目录被占用Device or resource busy
容易状态dead以及exit

Error response from daemon: Cannot start container XXX: Error getting container ddf1dd91bbf46dc648268327f8f7c6fffaf2f19cda5cf1d97fdc701016d4332c from driver devicemapper: Error mounting 目录': device or resource busy

删除、卸载目录均报错Device or resource busy：

rm -rf /docker-root/devicemapper/mnt/7d7200e04c5a166606b5bade185af33b7a9e85c9a0f49a3331eb58a0fdc4b4ae
umount /docker-root/devicemapper/mnt/7d7200e04c5a166606b5bade185af33b7a9e85c9a0f49a3331eb58a0fdc4b4ae

3. 解决办法

存在mount泄漏，编写脚本获取相应占用目录进程：

[root@host-centos7 deploy]# cat leak_rbd.sh 
#!/bin/bash
declare -A map
for i in `find /proc/*/mounts -exec grep $1 {} + 2>/dev/null | awk '{print $1"#"$2}'`
do
        pid=`echo $i | awk -F "[/]" '{print $3}'`
        point=`echo $i | awk -F "[#]" '{print $2}'`
        mnt=`ls -l /proc/$pid/ns/mnt |awk '{print $11}'`
        map["$mnt"]="exist"
        cmd=`cat /proc/$pid/cmdline`
        echo -e "$pid\t$mnt\t$cmd\t$point"
done

for i in `ps aux|grep docker-containerd-shim |grep -v "grep" |awk '{print $2}'`
do
        mnt=`ls -l /proc/$i/ns/mnt  2>/dev/null | awk '{print $11}'`
        if [[ "${map[$mnt]}" == "exist" ]];then
                ps aux |grep $i |grep -v "grep" |awk '{print $11" "$12}'
        fi
done

执行：

[root@host-centos7 deploy]# sh leak_rbd.sh /hel/docker-root/devicemapper/mnt/7d7200e04c5a166606b5bade185af33b7a9e85c9a0f49a3331eb58a0fdc4b4ae
**4647**	mnt:[4026536961]	/usr/sbin/ntpd-untp:ntp-g	/hel/docker-root/devicemapper/mnt/7d7200e04c5a166606b5bade185af33b7a9e85c9a0f49a3331eb58a0fdc4b4ae
[root@host-centos7 deploy]# ps -ef |grep 4647
ntp       **4647**     1  0 Aug08 ?        00:00:00 /usr/sbin/ntpd -u ntp:ntp -g
root     24886 11976  0 15:35 pts/1    00:00:00 grep --color=auto 4647

如上，发现是ntp服务占用目录，先停止该服务然后删除目录即可。

kill 4647
rm -rf /hel/docker-root/devicemapper/mnt/7d7200e04c5a166606b5bade185af33b7a9e85c9a0f49a3331eb58a0fdc4b4ae

启动docker服务正常！

4. 总结

有说是docker版本存储的bug，其他解决办法如网上说的docker.service中设置MountFlags，没有试验~