导语：在windows10的docker中启动非GPU应用容器会报错libnvidia-ml.so.1: file exists: unknown。但是这个镜像在其他linux环境可以跑。并且在wsl中运行mysql和zk容器是没问题。
以下是2个镜像的history。
10.10.3.5/cta/java-egl:01
IMAGE CREATED CREATED BY SIZE COMMENT
sha256:cf0a449929dbbeadd303ac4e4d9558bbb8ed35705ee0698b590c199f9da20c73 8 weeks ago /bin/sh -c #(nop) ENV PATH=/opt/jdk1.8.0_151/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 0B
<missing> 8 weeks ago /bin/sh -c #(nop) ENV CLASSPATH=.:/opt/jdk1.8.0_151/lib:/opt/jdk1.8.0_151/jre/lib 0B
<missing> 8 weeks ago /bin/sh -c #(nop) ENV JRE_HOME=/opt/jdk1.8.0_151/jre 0B
<missing> 8 weeks ago /bin/sh -c #(nop) ENV JAVA_HOME=/opt/jdk1.8.0_151 0B
<missing> 8 weeks ago /bin/sh -c #(nop) ENV LANG=C.UTF-8 0B
<missing> 8 weeks ago /bin/sh -c #(nop) ENV LC_ALL=C.UTF-8 0B
<missing> 8 weeks ago /bin/sh -c rm -rf /etc/localtime && ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && rm -rf /var/lib/apt/lists/* && rm -rf /root/.cache/pip/ 27.4kB
<missing> 8 weeks ago /bin/sh -c apt update && python3 /srv/get-pip.py -i https://pypi.tuna.tsinghua.edu.cn/simple && pip3 install gpustat -i https://pypi.tuna.tsinghua.edu.cn/simple 20.1MB
<missing> 8 weeks ago /bin/sh -c #(nop) ADD file:563e07b12e2bcf53a5a5111f4599069dd031cf8144f40abf3f8baa3274f9b6e3 in /srv/get-pip.py 2.16MB
<missing> 8 weeks ago /bin/sh -c echo "deb [trusted=yes] http://172.16.40.245:86 soft/" > /etc/apt/sources.list && rm -rf /etc/apt/sources.list.d/* && apt-get update && apt-get -y install python3.6 --no-install-recommends && rm -rf /usr/bin/python3 && ln -s /usr/bin/python3.6 /usr/bin/python3 && apt clean && rm -rf /var/lib/apt/lists/* 31.6MB
<missing> 2 months ago /bin/sh -c #(nop) ADD file:7d6c5b7a6e5c425195cac47638185cb9c7c3eb9c53603572635b38d3a61904e8 in /opt/ 386MB
<missing> 2 months ago /bin/sh -c #(nop) MAINTAINER xujiamin 0B
<missing> 4 months ago COPY usr /usr # buildkit 1.11kB buildkit.dockerfile.v0
<missing> 4 months ago COPY /usr/local/include /usr/local/include # buildkit 1.71MB buildkit.dockerfile.v0
<missing> 4 months ago RUN /bin/sh -c apt-get update && apt-get install -y --no-install-recommends pkg-config libxau-dev libxau-dev:i386 libxdmcp-dev libxdmcp-dev:i386 libxcb1-dev libxcb1-dev:i386 libxext-dev libxext-dev:i386 libx11-dev libx11-dev:i386 && rm -rf /var/lib/apt/lists/* # buildkit 67.2MB buildkit.dockerfile.v0
<missing> 4 months ago ENV LD_LIBRARY_PATH=/usr/local/lib/x86_64-linux-gnu:/usr/local/lib/i386-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/i386-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 0B buildkit.dockerfile.v0
<missing> 4 months ago RUN /bin/sh -c echo '/usr/local/lib/x86_64-linux-gnu' >> /etc/ld.so.conf.d/glvnd.conf && echo '/usr/local/lib/i386-linux-gnu' >> /etc/ld.so.conf.d/glvnd.conf && ldconfig # buildkit 20.9kB buildkit.dockerfile.v0
<missing> 4 months ago COPY 10_nvidia.json /usr/local/share/glvnd/egl_vendor.d/10_nvidia.json # buildkit 107B buildkit.dockerfile.v0
<missing> 4 months ago COPY /usr/local/lib/i386-linux-gnu /usr/local/lib/i386-linux-gnu # buildkit 1.11MB buildkit.dockerfile.v0
<missing> 4 months ago COPY /usr/local/lib/x86_64-linux-gnu /usr/local/lib/x86_64-linux-gnu # buildkit 1.71MB buildkit.dockerfile.v0
<missing> 4 months ago LABEL maintainer=NVIDIA CORPORATION <cudatools@nvidia.com> 0B buildkit.dockerfile.v0
<missing> 4 months ago ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/usr/lib/i386-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 0B buildkit.dockerfile.v0
<missing> 4 months ago COPY NGC-DL-CONTAINER-LICENSE / # buildkit 16kB buildkit.dockerfile.v0
<missing> 4 months ago RUN /bin/sh -c echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf # buildkit 46B buildkit.dockerfile.v0
<missing> 4 months ago ENV NVIDIA_DRIVER_CAPABILITIES=graphics,compat32,utility 0B buildkit.dockerfile.v0
<missing> 4 months ago ENV NVIDIA_VISIBLE_DEVICES=all 0B buildkit.dockerfile.v0
<missing> 4 months ago RUN /bin/sh -c dpkg --add-architecture i386 && apt-get update && apt-get install -y --no-install-recommends libxau6 libxau6:i386 libxdmcp6 libxdmcp6:i386 libxcb1 libxcb1:i386 libxext6 libxext6:i386 libx11-6 libx11-6:i386 && rm -rf /var/lib/apt/lists/* # buildkit 15.5MB buildkit.dockerfile.v0
<missing> 4 months ago LABEL maintainer=NVIDIA CORPORATION <cudatools@nvidia.com> 0B buildkit.dockerfile.v0
<missing> 8 months ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
<missing> 8 months ago /bin/sh -c mkdir -p /run/systemd && echo 'docker' > /run/systemd/container 7B
<missing> 8 months ago /bin/sh -c rm -rf /var/lib/apt/lists/* 0B
<missing> 8 months ago /bin/sh -c set -xe && echo '#!/bin/sh' > /usr/sbin/policy-rc.d && echo 'exit 101' >> /usr/sbin/policy-rc.d && chmod +x /usr/sbin/policy-rc.d && dpkg-divert --local --rename --add /sbin/initctl && cp -a /usr/sbin/policy-rc.d /sbin/initctl && sed -i 's/^exit.*/exit 0/' /sbin/initctl && echo 'force-unsafe-io' > /etc/dpkg/dpkg.cfg.d/docker-apt-speedup && echo 'DPkg::Post-Invoke { "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true"; };' > /etc/apt/apt.conf.d/docker-clean && echo 'APT::Update::Post-Invoke { "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true"; };' >> /etc/apt/apt.conf.d/docker-clean && echo 'Dir::Cache::pkgcache ""; Dir::Cache::srcpkgcache "";' >> /etc/apt/apt.conf.d/docker-clean && echo 'Acquire::Languages "none";' > /etc/apt/apt.conf.d/docker-no-languages && echo 'Acquire::GzipIndexes "true"; Acquire::CompressionTypes::Order:: "gz";' > /etc/apt/apt.conf.d/docker-gzip-indexes && echo 'Apt::AutoRemove::SuggestsImportant "false";' > /etc/apt/apt.conf.d/docker-autoremove-suggests 745B
<missing> 8 months ago /bin/sh -c #(nop) ADD file:11b425d4c08e81a3e0cb2e0345d27cd5fc844dd83f1096af4cc05f635824ff5d in / 135MB
egl-tedt:0508
IMAGE CREATED CREATED BY SIZE COMMENT
sha256:0814263078c7682d0101149f620bb690623c5eabf6b50863be1cea78c773ac74 7 minutes ago /bin/sh -c #(nop) ENV PATH=/opt/jdk1.8.0_151/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 0B
sha256:adb02445b5295373cd80a45d9bb04cd7dbe33bcc878f79a71e5b7ca84f571054 7 minutes ago /bin/sh -c #(nop) ENV CLASSPATH=.:/opt/jdk1.8.0_151/lib:/opt/jdk1.8.0_151/jre/lib 0B
sha256:ab4c20cea47fb6f48ad77b4daf9fd6c8b680131bdadc3b14b96f61900a7cdcf6 7 minutes ago /bin/sh -c #(nop) ENV JRE_HOME=/opt/jdk1.8.0_151/jre 0B
sha256:0b69e576d8488802f1a15c95d5dfd0b7ae7742fa867cf2cc2e23bd2389566454 7 minutes ago /bin/sh -c #(nop) ENV JAVA_HOME=/opt/jdk1.8.0_151 0B
sha256:a56b777a5b288a115ba9dd32f0c506c3206cb684a3758487f06a51dd25ef1304 7 minutes ago /bin/sh -c #(nop) ENV LANG=C.UTF-8 0B
sha256:5252ea054dd647ac1396080e22fab9678efe388b255e987c52ace139d2730282 7 minutes ago /bin/sh -c #(nop) ENV LC_ALL=C.UTF-8 0B
sha256:2d68568286bd14508ab66faa9c16f0d671b33f89b7d3e10e85eebb17ae94aac6 7 minutes ago /bin/sh -c rm -rf /etc/localtime && ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && rm -rf /var/lib/apt/lists/* && rm -rf /root/.cache/pip/ 33B
sha256:5ac15cb26510ce01c71245f5437af6aede36f8cc110aff48de0a36b58bdde336 7 minutes ago /bin/sh -c apt update && python3 /srv/get-pip.py -i https://pypi.tuna.tsinghua.edu.cn/simple && pip3 install gpustat -i https://pypi.tuna.tsinghua.edu.cn/simple 20.1MB
sha256:c2d0acc9bf7d90a69982ea8edbcb24703cb246bbf07f03ec2286162ad04b0634 8 minutes ago /bin/sh -c #(nop) ADD file:563e07b12e2bcf53a5a5111f4599069dd031cf8144f40abf3f8baa3274f9b6e3 in /srv/get-pip.py 2.16MB
sha256:13e907c54f100eb777b52e0b182d0f737882948aec599457044a2610ac071edb 8 minutes ago /bin/sh -c echo "deb [trusted=yes] http://172.16.40.245:86 soft/" > /etc/apt/sources.list && rm -rf /etc/apt/sources.list.d/* && apt-get update && apt-get -y install python3.6 --no-install-recommends && rm -rf /usr/bin/python3 && ln -s /usr/bin/python3.6 /usr/bin/python3 && apt clean && rm -rf /var/lib/apt/lists/* 31.6MB
sha256:003a38a90ba972418f33ed5f71243b43a04e62d308f8e3a476e289da42ee8c08 8 minutes ago /bin/sh -c #(nop) ADD file:7d6c5b7a6e5c425195cac47638185cb9c7c3eb9c53603572635b38d3a61904e8 in /opt/ 386MB
sha256:934a84a92741443b3af9fb6c71ff3f44d172a1c163fb81fece9fb264a07a1505 2 days ago /bin/sh -c #(nop) MAINTAINER xujiamin 0B
sha256:4d58112735c070147b17dc51e1e8cf91a3970ad3f026f7535bc15a961fbdeaa9 4 months ago COPY usr /usr # buildkit 1.11kB buildkit.dockerfile.v0
<missing> 4 months ago COPY /usr/local/include /usr/local/include # buildkit 1.71MB buildkit.dockerfile.v0
<missing> 4 months ago RUN /bin/sh -c apt-get update && apt-get install -y --no-install-recommends pkg-config libxau-dev libxau-dev:i386 libxdmcp-dev libxdmcp-dev:i386 libxcb1-dev libxcb1-dev:i386 libxext-dev libxext-dev:i386 libx11-dev libx11-dev:i386 && rm -rf /var/lib/apt/lists/* # buildkit 67.2MB buildkit.dockerfile.v0
<missing> 4 months ago ENV LD_LIBRARY_PATH=/usr/local/lib/x86_64-linux-gnu:/usr/local/lib/i386-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/i386-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 0B buildkit.dockerfile.v0
<missing> 4 months ago RUN /bin/sh -c echo '/usr/local/lib/x86_64-linux-gnu' >> /etc/ld.so.conf.d/glvnd.conf && echo '/usr/local/lib/i386-linux-gnu' >> /etc/ld.so.conf.d/glvnd.conf && ldconfig # buildkit 20.9kB buildkit.dockerfile.v0
<missing> 4 months ago COPY 10_nvidia.json /usr/local/share/glvnd/egl_vendor.d/10_nvidia.json # buildkit 107B buildkit.dockerfile.v0
<missing> 4 months ago COPY /usr/local/lib/i386-linux-gnu /usr/local/lib/i386-linux-gnu # buildkit 1.11MB buildkit.dockerfile.v0
<missing> 4 months ago COPY /usr/local/lib/x86_64-linux-gnu /usr/local/lib/x86_64-linux-gnu # buildkit 1.71MB buildkit.dockerfile.v0
<missing> 4 months ago LABEL maintainer=NVIDIA CORPORATION <cudatools@nvidia.com> 0B buildkit.dockerfile.v0
<missing> 4 months ago ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:/usr/lib/i386-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 0B buildkit.dockerfile.v0
<missing> 4 months ago COPY NGC-DL-CONTAINER-LICENSE / # buildkit 16kB buildkit.dockerfile.v0
<missing> 4 months ago RUN /bin/sh -c echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf # buildkit 46B buildkit.dockerfile.v0
<missing> 4 months ago ENV NVIDIA_DRIVER_CAPABILITIES=graphics,compat32,utility 0B buildkit.dockerfile.v0
<missing> 4 months ago ENV NVIDIA_VISIBLE_DEVICES=all 0B buildkit.dockerfile.v0
<missing> 4 months ago RUN /bin/sh -c dpkg --add-architecture i386 && apt-get update && apt-get install -y --no-install-recommends libxau6 libxau6:i386 libxdmcp6 libxdmcp6:i386 libxcb1 libxcb1:i386 libxext6 libxext6:i386 libx11-6 libx11-6:i386 && rm -rf /var/lib/apt/lists/* # buildkit 15.5MB buildkit.dockerfile.v0
<missing> 4 months ago LABEL maintainer=NVIDIA CORPORATION <cudatools@nvidia.com> 0B buildkit.dockerfile.v0
<missing> 8 months ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
<missing> 8 months ago /bin/sh -c mkdir -p /run/systemd && echo 'docker' > /run/systemd/container 7B
<missing> 8 months ago /bin/sh -c rm -rf /var/lib/apt/lists/* 0B
<missing> 8 months ago /bin/sh -c set -xe && echo '#!/bin/sh' > /usr/sbin/policy-rc.d && echo 'exit 101' >> /usr/sbin/policy-rc.d && chmod +x /usr/sbin/policy-rc.d && dpkg-divert --local --rename --add /sbin/initctl && cp -a /usr/sbin/policy-rc.d /sbin/initctl && sed -i 's/^exit.*/exit 0/' /sbin/initctl && echo 'force-unsafe-io' > /etc/dpkg/dpkg.cfg.d/docker-apt-speedup && echo 'DPkg::Post-Invoke { "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true"; };' > /etc/apt/apt.conf.d/docker-clean && echo 'APT::Update::Post-Invoke { "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true"; };' >> /etc/apt/apt.conf.d/docker-clean && echo 'Dir::Cache::pkgcache ""; Dir::Cache::srcpkgcache "";' >> /etc/apt/apt.conf.d/docker-clean && echo 'Acquire::Languages "none";' > /etc/apt/apt.conf.d/docker-no-languages && echo 'Acquire::GzipIndexes "true"; Acquire::CompressionTypes::Order:: "gz";' > /etc/apt/apt.conf.d/docker-gzip-indexes && echo 'Apt::AutoRemove::SuggestsImportant "false";' > /etc/apt/apt.conf.d/docker-autoremove-suggests 745B
<missing> 8 months ago /bin/sh -c #(nop) ADD file:11b425d4c08e81a3e0cb2e0345d27cd5fc844dd83f1096af4cc05f635824ff5d in / 135MB
通过命令docker history image --no-trunc对比发现2个镜像的构建的命令和文件都是相同的。
egl-test:0508是我在一台没有显卡且没有安装显卡驱动的机子上重新制作的镜像。
10.10.3.5/cta/java-egl:01 是我很早之前做的镜像。 2个镜像的Dockerfile是同一个，但是对应的显卡驱动却不一样。
把Dockerfile文件夹拷贝到一台已经安装了显卡驱动的机子上测试。离谱 居然复现了这个问题
测试使用–no-cache 构建镜像也是一样的问题。
怀疑是不是docker的问题 。修改daemon.json
原daemon.json
{
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"path": "/usr/bin/nvidia-container-runtime",
"runtimeArgs": []
}
},
"default-shm-size": "2G",
"insecure-registries": ["harbor.deepwise.com","10.10.3.5","172.28.3.5"] ,
"graph":"/data1/docker/lib/docker"
}
修改后
{
"graph":"/data1/docker/lib/docker"
}
解决。刚刚还会出现的nvidia相关信息消失了。
查阅资料 也有人有类似问题
可参考对方的不从头构建镜像的解决办法，需要修改daemon.json，否则修改的时候会报device or resource busy。
FROM <the image you care about>
RUN umount /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1 && \
rm -rf /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1
...
我验证的是这个方式
FROM <the image you care about>
RUN rm -rf /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1 /usr/lib/x86_64-linux-gnu/libcuda.so.1
参考issue
https://github.com/NVIDIA/nvidia-docker/issues/1551
342
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
