CentOS下安裝Nvidia docker 2.0:[Errno 256] No more mirrors to try錯誤及解決方式

前言

這個錯誤是筆者在為CentOS 7下安裝nvidia-docker 2.0版時所出現的。

發生背景

首先照著NVIDIA/nvidia-docker Installation (version 2.0)的教學移除nvidia-docker 1.0版。
然後參考Repository configuration，根據Linux版本來安裝對應的倉庫。

使用的指令如下：

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | \
  sudo tee /etc/yum.repos.d/nvidia-docker.repo

然後

DIST=$(sed -n 's/releasever=//p' /etc/yum.conf)
DIST=${DIST:-$(. /etc/os-release; echo $VERSION_ID)}
sudo rpm -e gpg-pubkey-f796ecb0
sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/$DIST/nvidia-docker/gpgdir --delete-key f796ecb0
sudo yum makecache

但是在執行sudo yum makecache時就出現了以下的錯誤。

錯誤訊息

Geladene Plugins: fastestmirror
base | 3.6 kB 00:00:00
cuda-8-0-local-ga2 | 2.5 kB 00:00:00
cuda-9-1-local | 2.5 kB 00:00:00
docker-ce-stable | 3.5 kB 00:00:00
epel | 3.2 kB 00:00:00
extras | 3.4 kB 00:00:00
libnvidia-container/x86_64/signature | 455 B 00:00:00
Schlüssel wird von https://nvidia.github.io/libnvidia-container/gpgkey geholt
libnvidia-container/x86_64/signature | 2.0 kB 00:00:00 !!!
https://nvidia.github.io/libnvidia-container/centos7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for libnvidia-container
Anderer Spiegelserver wird versucht.
nvidia-container-runtime/x86_64/signature | 455 B 00:00:00
Schlüssel wird von https://nvidia.github.io/nvidia-container-runtime/gpgkey geholt
nvidia-container-runtime/x86_64/signature | 2.0 kB 00:00:00 !!!
https://nvidia.github.io/nvidia-container-runtime/centos7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for nvidia-container-runtime
Anderer Spiegelserver wird versucht.
nvidia-docker/x86_64/signature | 455 B 00:00:00
Schlüssel wird von https://nvidia.github.io/nvidia-docker/gpgkey geholt
GPG-Schlüssel 0xF796ECB0 importieren:
Benutzerkennung : “NVIDIA CORPORATION (Open Source Projects) cudatools@nvidia.com”
Fingerabdruck: c95b 321b 61e8 8c18 09c4 f759 ddca e044 f796 ecb0
Von : https://nvidia.github.io/nvidia-docker/gpgkey
Ist dies in Ordnung? [j/N] :j
nvidia-docker/x86_64/signature | 2.0 kB 00:00:02 !!!
updates | 3.4 kB 00:00:00
(1/21): cuda-8-0-local-ga2/other_db | 4.5 kB 00:00:00
(2/21): cuda-9-1-local/filelists_db | 78 kB 00:00:00
(3/21): cuda-8-0-local-ga2/filelists_db | 71 kB 00:00:00
(4/21): cuda-9-1-local/other_db | 5.4 kB 00:00:00
(5/21): docker-ce-stable/x86_64/filelists_db | 10 kB 00:00:01
(6/21): docker-ce-stable/x86_64/other_db | 107 kB 00:00:01
(7/21): epel/x86_64/updateinfo | 934 kB 00:00:05
(8/21): epel/x86_64/prestodelta | 1.3 kB 00:00:00
(9/21): base/7/x86_64/other_db | 2.5 MB 00:00:19
(10/21): epel/x86_64/primary | 3.6 MB 00:00:27
(11/21): extras/7/x86_64/filelists_db | 603 kB 00:00:02
(12/21): extras/7/x86_64/prestodelta | 100 kB 00:00:02
(13/21): extras/7/x86_64/other_db | 126 kB 00:00:00
(14/21): nvidia-docker/x86_64/primary | 4.4 kB 00:00:00
(15/21): nvidia-docker/x86_64/other | 3.6 kB 00:00:00
(16/21): nvidia-docker/x86_64/filelists | 2.8 kB 00:00:01
(17/21): epel/x86_64/other | 2.2 MB 00:00:06
(18/21): updates/7/x86_64/other_db | 578 kB 00:00:03
(19/21): updates/7/x86_64/filelists_db | 3.4 MB 00:00:07
(20/21): epel/x86_64/filelists | 10 MB 00:00:59
(21/21): base/7/x86_64/filelists_db | 6.9 MB 00:01:12
Loading mirror speeds from cached hostfile
*base: free.nchc.org.tw
*extras: free.nchc.org.tw
*updates: free.nchc.org.tw
epel 12724/12724
epel 12724/12724
epel 12724/12724
https://nvidia.github.io/libnvidia-container/centos7/x86_64/repodata/dfe68d28fb587013c513050804570d27a4afa8104da395bdf795eb232c55719c89554627972f78d0c7fabb372f2c219e6f54e12e73d77e825853a880d98800f3-filelists.xml.xz: [Errno 14] HTTPS Error 404 - Not Found
Anderer Spiegelserver wird versucht.
To address this issue please refer to the below wiki article
https://wiki.centos.org/yum-errors
If above article doesn’t help to resolve this issue please use https://bugs.centos.org/.
One of the configured repositories failed (libnvidia-container),
and yum doesn’t have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work “fix” this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Run the command with the repository temporarily disabled
yum --disablerepo=libnvidia-container …
4. Disable the repository permanently, so yum won’t use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:
yum-config-manager --disable libnvidia-container
or
subscription-manager repos --disable=libnvidia-container
5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=libnvidia-container.skip_if_unavailable=true
failure: repodata/dfe68d28fb587013c513050804570d27a4afa8104da395bdf795eb232c55719c89554627972f78d0c7fabb372f2c219e6f54e12e73d77e825853a880d98800f3-filelists.xml.xz from libnvidia-container: [Errno 256] No more mirrors to try.
https://nvidia.github.io/libnvidia-container/centos7/x86_64/repodata/dfe68d28fb587013c513050804570d27a4afa8104da395bdf795eb232c55719c89554627972f78d0c7fabb372f2c219e6f54e12e73d77e825853a880d98800f3-filelists.xml.xz: [Errno 14] HTTPS Error 404 - Not Found

解決辦法

首先參考yum-errors及[Errno 14] HTTP Error 404 - Not Found上的建議，將機器上的快取給清除：

yum clean all
rm -rf /var/cache/yum/*

然後試著執行

yum update -y

卻會出現相似的錯誤：

Geladene Plugins: fastestmirror
base | 3.6 kB 00:00:00
cuda-8-0-local-ga2 | 2.5 kB 00:00:00
cuda-9-1-local | 2.5 kB 00:00:00
docker-ce-stable | 3.5 kB 00:00:00
epel | 3.2 kB 00:00:00
extras | 3.4 kB 00:00:00
libnvidia-container/x86_64/signature | 455 B 00:00:00
Schlüssel wird von https://nvidia.github.io/libnvidia-container/gpgkey geholt
libnvidia-container/x86_64/signature | 2.0 kB 00:00:00 !!!
https://nvidia.github.io/libnvidia-container/centos7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for libnvidia-container
Anderer Spiegelserver wird versucht.
One of the configured repositories failed (libnvidia-container),
and yum doesn’t have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work “fix” this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Run the command with the repository temporarily disabled
yum --disablerepo=libnvidia-container …
4. Disable the repository permanently, so yum won’t use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:
yum-config-manager --disable libnvidia-container
or
subscription-manager repos --disable=libnvidia-container
5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=libnvidia-container.skip_if_unavailable=true
failure: repodata/repomd.xml from libnvidia-container: [Errno 256] No more mirrors to try.
https://nvidia.github.io/libnvidia-container/centos7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for libnvidia-container

這代表問題並沒有解決。

後來用錯誤訊息中的libnvidia-container這個關鍵字去搜尋，才發現找到Github上的這則issue：centos7 broken dependencies with 3:18.09。

其中flx42建議使用gpg來將nvidia-docker, nvidia-container-runtime , libnvidia-container這三個倉庫全部移除（教學中僅移除nvidia-docker，然而這並不夠）：

sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/$DIST/nvidia-docker/gpgdir --delete-key f796ecb0
sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/$DIST/nvidia-container-runtime/gpgdir --delete-key f796ecb0
sudo gpg --homedir /var/lib/yum/repos/$(uname -m)/$DIST/libnvidia-container/gpgdir --delete-key f796ecb0

以下為三個指令的輸出訊息：

gpg: WARNUNG: Unsichere Zugriffsrechte des Home-Verzeichnis `/var/lib/yum/repos/x86_64/7/nvidia-docker/gpgdir’
gpg (GnuPG) 2.0.22; Copyright © 2013 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
pub 4096R/F796ECB0 2017-09-28 NVIDIA CORPORATION (Open Source Projects) cudatools@nvidia.com
Diesen Schlüssel aus dem Schlüsselbund löschen? (j/N) j

gpg: WARNUNG: Unsichere Zugriffsrechte des Home-Verzeichnis `/var/lib/yum/repos/x86_64/7/nvidia-container-runtime/gpgdir’
gpg (GnuPG) 2.0.22; Copyright © 2013 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
pub 4096R/F796ECB0 2017-09-28 NVIDIA CORPORATION (Open Source Projects) cudatools@nvidia.com
Diesen Schlüssel aus dem Schlüsselbund löschen? (j/N) j

gpg: WARNUNG: Unsichere Zugriffsrechte des Home-Verzeichnis `/var/lib/yum/repos/x86_64/7/libnvidia-container/gpgdir’
gpg (GnuPG) 2.0.22; Copyright © 2013 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
pub 4096R/F796ECB0 2017-09-28 NVIDIA CORPORATION (Open Source Projects) cudatools@nvidia.com
Diesen Schlüssel aus dem Schlüsselbund löschen? (j/N) j

再次執行sudo yum makecache -y，就發現問題己成功解決！

發生原因

問題己經解決了，現在我們再頭看看到底發生錯誤的原因是什麼？

回顧Repository configuration這個頁面，當中有一段：

In order to update the nvidia-docker repository key for your distribution, follow the instructions below.

說明為了要更新nvidia-docker倉庫的key，要先使用rpm -e來刪除gpg-pubkey-f796ecb0這個package。
然後用gpg --delete-key來移除/var/lib/yum/repos/$(uname -m)/$DIST/nvidia-container-runtime/gpgdir目錄下名為f796ecb0的public key。

再回顧錯誤訊息，當中有一段：

[Errno -1] repomd.xml signature could not be verified for libnvidia-container

它說明了本地的key無法通過認證，因此解決方法就是把對應的repo的public key給刪除。

參考連結

NVIDIA/nvidia-docker Installation (version 2.0)
Repository configuration
yum-errors
[Errno 14] HTTP Error 404 - Not Found
centos7 broken dependencies with 3:18.09
wangruohui/Install NVIDIA Driver and CUDA.md