NVIDIA-SMI has failed because it couldn‘t communicate with the NVIDIA driver.

来源：nvidia

问题

NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running

解决

Even with those commands, the issue wasn’t solved.
Eventually, the fastest way to fix 2 machines with a package manager is to purge all Nvidia & Cuda,did it by:

sudo apt-get remove --purge '^nvidia-.*'
sudo apt-get remove --purge '^libnvidia-.*'
sudo apt-get remove --purge '^cuda-.*'

Then after it’s clean ran that:

sudo apt-get install linux-headers-$(uname -r)

From here - it’s the same for all VMs:
Download latest run file from Nvidia site, and run it, accept if needed to upgrade current, or install from scratch.
The driver is back to work.

The issue was started after did some updates, and the Linux kernel was changed.