ERROR: No supported GPU(s) detected to run this container

ERROR: No supported GPU(s) detected to run this container

在拉取yolov5官方Docker image之后docker run --ipc=host --gpus all -it ultralytics/yolov5:latest启动容器
使用以下命令查看GPU：

root@87fb0c80ec7f:~# nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Nov_30_19:08:53_PST_2020
Cuda compilation tools, release 11.2, V11.2.67
Build cuda_11.2.r11.2/compiler.29373293_0

root@87fb0c80ec7f:~# nvidia-smi -L
GPU 0: GeForce RTX 3090 (UUID: GPU-834af3c0-5d3b-00a7-c027-e68486b35d2d)

root@87fb0c80ec7f:~# nvidia-smi 
Tue Mar  9 06:41:13 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.39       Driver Version: 460.39       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090    Off  | 00000000:65:00.0  On |                  N/A |
| 34%   34C    P8    29W / 350W |    242MiB / 24265MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

以上均可以正常查看GPU相关信息，但是在使用的时候却发现用不了GPU。

root@87fb0c80ec7f:~# python -c "import torch; print(torch.cuda.is_available())"
/opt/conda/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at  ../c10/cuda/CUDAFunctions.cpp:104.)
  return torch._C._cuda_getDeviceCount() > 0
False

回头查看前面启动容器之后的打印信息：

ERROR: No supported GPU(s) detected to run this container

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

相关issue显示：

可能是Docker安装有问题，或者是CUDA安装有问题

小结：
在容器中，即使nvcc -V，nvidia-smi都表现正常，GPU还是有用不了的可能

搞了一天，突然想起会不会是GPU设备被占用了，遂将电脑重启，果然重启大法好。