【报错】UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount().

问题

服务器重启后，使用显卡的程序突然不能运行了，运行torch.cuda.is_available()报标题的错误。

具体如下：

>>> import torch
>>> torch.cuda
<module 'torch.cuda' from '/home/gpu-server/anaconda3/lib/python3.7/site-packages/torch/cuda/__init__.py'>
>>> torch.cuda.is_available
<function is_available at 0x7fb8f79017a0>
>>> torch.cuda.is_available()
/home/gpu-server/anaconda3/lib/python3.7/site-packages/torch/cuda/__init__.py:88: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 101: invalid device ordinal (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() > 0
False
>>> torch.cuda.device_count
<functools._lru_cache_wrapper object at 0x7fb8f783fd70>
>>> torch.cuda.device_count()
7

解决

sudo reboot

参考：https://stackoverflow.com/questions/66371130/cuda-initialization-unexpected-error-from-cudagetdevicecount