NVIDIA-SMI has failed because it couldn‘t communicate with the NVIDIA driver.

blog
docker等系统操作遇到的问题
ubuntu20.04[centos7、win类似]记录

1 centos遇到解决办法

之前centos遇到这问题，解决方法

2 ubuntu遇到解决办法

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

网上的介绍大多都不可行，这里找到一个方法就是重新卸载nvidia驱动和cuda，重装即可。用run格式的很好装。

---

2.1 卸载一切

# 卸载nvidia和cuda相关
sudo apt-get remove cuda
sudo apt-get autoclean
sudo apt-get remove cuda* nvidia*
cd /usr/local
sudo rm -rvf cuda*

2.2 下载cuda和cudnn

然后下载run格式的cuda。
cuda下载地址
cudnn下载地址
cuda_10.2.89_440.33.01_linux.run
cudnn-10.2-linux-x64-v7.6.5.32.tgz，记得别忘记了下载和安装这

2.3 安装驱动

为安全起见，先装驱动，再去装cuda

sudo chmod 755 cuda_10.2.89_440.33.01_linux.run
sudo ./cuda_10.2.89_440.33.01_linux.run

【直接输入accept】敲回车,【只选择驱动】【Install】即可。安装完成后用nvidia-smi去检查一下

===========
= Summary =
===========

Driver:   Installed
Toolkit:  Not Selected
Samples:  Not Selected

To uninstall the NVIDIA Driver, run nvidia-uninstall
Logfile is /var/log/cuda-installer.log

2.4 安装cuda

===========
= Summary =
===========

Driver:   Not Selected
Toolkit:  Installed in /usr/local/cuda-10.2/
Samples:  Installed in /root/, but missing recommended libraries

Please make sure that
 -   PATH includes /usr/local/cuda-10.2/bin
 -   LD_LIBRARY_PATH includes /usr/local/cuda-10.2/lib64, or, add /usr/local/cuda-10.2/lib64 to /etc/ld.so.conf and run ldconfig as root

To uninstall the CUDA Toolkit, run cuda-uninstaller in /usr/local/cuda-10.2/bin

Please see CUDA_Installation_Guide_Linux.pdf in /usr/local/cuda-10.2/doc/pdf for detailed information on setting up CUDA.
***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 440.00 is required for CUDA 10.2 functionality to work.
To install the driver using this installer, run the following command, replacing <CudaInstaller> with the name of this run file:
    sudo <CudaInstaller>.run --silent --driver

Logfile is /var/log/cuda-installer.log

2.5 安装cudnn

tar xvzf cudnn-10.2-linux-x64-v7.6.5.32.tgz
sudo cp cuda/include/cudnn.h /usr/local/cuda-10.2/include/
sudo cp -r cuda/lib64/libcudnn* /usr/local/cuda-10.2/lib64/

2.6 配置环境变量

vim ~/.bashrc
#配置以下值即可
export PATH="/usr/local/cuda/bin:$PATH"
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64"
export CUDA_HOME=/usr/local/cuda