GPU无程序运行,仍有显存被占用,如图:
root@44668fac5caf:/CUDA# nvidia-smi
Fri Dec 6 05:25:17 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.74 Driver Version: 418.74 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P100-PCIE... Off | 00000000:3B:00.0 Off | 0 |
| N/A 42C P0 35W / 250W | 5338MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla P100-PCIE... Off | 00000000:AF:00.0 Off | N/A |
| N/A 41C P0 35W / 250W | 8241MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
解决:
通过下面命令查看GPU上所有的程序:
fuser -v /dev/nvidia*
若出现 fuser:command not found,可阅读这篇博客:
Ubuntu:fuser command not found
命令输出结果如下:
root@44668fac5caf:/CUDA# fuser -v /dev/nvidia*
USER PID ACCESS COMMAND
/dev/nvidia-uvm: root kernel mount /dev/nvidia-uvm
root 202377 F...m cuda
root 202562 F...m cuda
/dev/nvidia-uvm-tools:
root kernel mount /dev/nvidia-uvm-tools
/dev/nvidia0: root kernel mount /dev/nvidia0
root 1037 F.... nvidia-smi
root 202377 F...m cuda
root 202562 F...m cuda
/dev/nvidia1: root kernel mount /dev/nvidia1
root 202377 F...m cuda
root 202562 F...m cuda
/dev/nvidiactl: root kernel mount /dev/nvidiactl
root 1037 F.... nvidia-smi
root 202377 F...m cuda
root 202562 F...m cuda
可以按照下面命令,杀死相关进程:
kill -9 PID