Failed to initialize NVML: Driver/library version mismatch

Failed to initialize NVML: Driver/library version mismatch

问题：

the driver was not installed correctly. This can happen if the previous driver was installed using the runfile installer and the new driver was installed using package manager, or vice versa. There are probably other scenarios as well.

Remove all previous package manager installs, and all previous runfile installer installs, then reinstall the driver.

我们之前安装了.run文件的cuda和nvidia驱动。之后又利用apt命令安装了nvidia-cuda-toolkit和cuda。导致版本冲突，驱动不匹配问题。

卸载：

卸载cuda

卸载通过.run文件安装的cuda：

cd /usr/local/cuda-xx.x/bin/
sudo ./cuda-uninstaller
sudo rm -rf /usr/local/cuda-xx.x

卸载通过apt命令安装的cuda：

sudo apt-get remove "cuda*" "*cublas*" "*cufft*" "*curand*" "*cusolver*" "*cusparse*" "*npp*" "*nvjpeg*" "nsight*"

通过dpkg查找对应的package是否删除干净：

dpkg -l

查找对应版本，我这边装的9.1.85。通过版本确认已经删除干净。

卸载nvidia

卸载通过.run文件安装的nvidia驱动：

sudo /usr/bin/nvidia-uninstall

卸载之前安装的所有驱动，包括通过apt安装的：

sudo apt-get --purge remove "*nvidia*"

安装

安装cuda和nvidia驱动可以参考：

Ubuntu服务器安装nvidia-430.64、cuda-10.1,cudnn-7.6.0和anaconda

参考

当然也有些其他人遇到了相同的问题，采用的解决方式不一样可以作为参考：

NVIDIA NVML Driver/library version mismatch [closed]

nvidia-smi返回错误信息‘Failed to initialize NVML: Driver/library version mismatch’

官方提供了遇到冲突时的解决方案：
Handle Conflicting Installation Methods

官方卸载cuda和nvidia(runfile文件)的方式：
Uninstallation