解决GPU显存句柄泄漏问题

已于 2023-11-24 10:23:49 修改
阅读量347 收藏
点赞数
文章标签： 人工智能
于 2023-10-19 13:40:45 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/haronchou/article/details/133922013
版权
  1. cudastream资源没释放：释放了就小了很多
  2. 其他资源没释放，问题未知！！！模型资源释放了！！！
  3. 多线程导致的没释放完成！！！！

https://blog.csdn.net/liu_feng_zi_/article/details/103421821

现象是：/dev/nvidia0

1. 安装nvtop随时查看GPU情况 

git remote -v # 查看仓库从哪里拉取的

git clone https://github.com/Syllo/nvtop.git
mkdir -p nvtop/build && cd nvtop/build
cmake .. -DNVIDIA_SUPPORT=ON -DAMDGPU_SUPPORT=ON -DINTEL_SUPPORT=ON
make

# Install globally on the system
sudo make install
watch -n 0.1 nvidia-smi # 每隔0.1s查看nvidia-smi情况

ls -lh /proc/进程号/fd
ls -l /proc/进程号/fd

ps -T -p <pid>  或 
top -H -p <pid>

# 方法一：通过名字直接查看top
top -H -p $(pidof name)
top -H -p $(pgrep -f name)

# 方法二：通过程序名字查找进程号 progress id == pid 将返回进程号
ps -C <pro_name> -o pid=



# 看显存
fuser -v /dev/nvidia*

# 这个pro在当前文件夹下进行，要包含所有的库，cuda tensorrt opencv等，全部要包含，为了测试内存泄漏。测试文件在：Testlib文件夹下
cuda-memcheck --tool memcheck --leak-check full pro

# 结果：cuda-memcheck没有任何问题

# valgrind查看CPU内存泄漏
valgrind --leak-check=full ./pro

linux /proc/[pid]/fd 中socket描述符后面的数字是什么意思？inode（information node）_/proc/[pid]/fd inode定位文件位置-CSDN博客

1. 查看CPU内存泄漏

  • 在docker中安装valgrind 教程：valgrind基本功能介绍、基础使用方法说明-CSDN博客
  • 在Ubuntu主机中安装了valgrind，由于环境都在docker中，在docker下运行，是没有valgrind命令的。需要安装
  • 由于有了目录映射，就在外面下载好；在docker中执行如下命令：
    • 然后docker命令行下make install：就有了valgrind命令了！！！ make install
1、tar –xf valgrind-3.17.0.tar.bz2
2、cd valgrind-3.17.0
3、./configure         // 运行配置脚本生成makefile文件，可以--help查看配置项，自行按需配置，比如修改编译工具、修改安装路径等
4、make
5、make install        //安装生成可执行文件，可执行文件的路径有参数--prefix指定，需要在PATH中添加环境变量；若不加参数--prefix指定，仅使用默认配置，则会自动关联
  • 同前面的cuda-memcheck一样，需要把所有的运行库都放在pro所在的文件夹；（之前运行过cuda-memcheck，所以这里直接运行成功了） 
valgrind --leak-check=full ./pro
HEAP SUMMARY:
==22248==     in use at exit: 1,855,624 bytes in 4,645 blocks
==22248==   total heap usage: 11,181,264 allocs, 11,176,619 frees, 39,958,084,517 bytes allocated
==22248== 
==22248== 72 bytes in 1 blocks are possibly lost in loss record 733 of 1,502
==22248==    at 0x483C7F5: malloc (vg_replace_malloc.c:442)
==22248==    by 0x36BB8690: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x369F192A: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA522C: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA88BF: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FBE7FF: cudaSetDevice (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248== 
==22248== 72 bytes in 1 blocks are possibly lost in loss record 734 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x369DFDCA: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x369E6A19: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x369E7293: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA49ED: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA88BF: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248== 
==22248== 152 bytes in 1 blocks are possibly lost in loss record 1,327 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x36A52F94: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA48C4: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA88BF: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FBE7FF: cudaSetDevice (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x53257E1: set_device (simple_yolo.cu:334)
==22248==    by 0x53257E1: SimpleYolo::set_device(int) (simple_yolo.cu:330)
==22248== 
==22248== 152 bytes in 1 blocks are possibly lost in loss record 1,328 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x36A52F94: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA4F93: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA88BF: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FBE7FF: cudaSetDevice (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x53257E1: set_device (simple_yolo.cu:334)
==22248==    by 0x53257E1: SimpleYolo::set_device(int) (simple_yolo.cu:330)
==22248== 
==22248== 152 bytes in 1 blocks are possibly lost in loss record 1,329 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x36A52F94: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BC0DCC: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BC22F9: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BC2444: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA40B9: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248== 
==22248== 152 bytes in 1 blocks are possibly lost in loss record 1,330 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x36A52F94: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BC0DCC: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BC22F9: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BC2467: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA40B9: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248== 
==22248== 152 bytes in 1 blocks are possibly lost in loss record 1,331 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x36A52F94: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA4239: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA88BF: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FBE7FF: cudaSetDevice (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x53257E1: set_device (simple_yolo.cu:334)
==22248==    by 0x53257E1: SimpleYolo::set_device(int) (simple_yolo.cu:330)
==22248== 
==22248== 152 bytes in 1 blocks are possibly lost in loss record 1,332 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x36A52F94: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x369E66B9: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x369E72AF: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA49ED: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA88BF: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248== 
==22248== 152 bytes in 1 blocks are possibly lost in loss record 1,333 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x36A52F94: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA4AAC: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA88BF: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FBE7FF: cudaSetDevice (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x53257E1: set_device (simple_yolo.cu:334)
==22248==    by 0x53257E1: SimpleYolo::set_device(int) (simple_yolo.cu:330)
==22248== 
==22248== 152 bytes in 1 blocks are possibly lost in loss record 1,334 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x36A52F94: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA4C4E: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA88BF: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FBE7FF: cudaSetDevice (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x53257E1: set_device (simple_yolo.cu:334)
==22248==    by 0x53257E1: SimpleYolo::set_device(int) (simple_yolo.cu:330)
==22248== 
==22248== 152 bytes in 1 blocks are possibly lost in loss record 1,335 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x36A52F94: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x368DD177: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA4CB4: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA88BF: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FBE7FF: cudaSetDevice (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248== 
==22248== 152 bytes in 1 blocks are possibly lost in loss record 1,336 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x36A52F94: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x368DD197: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA4CB4: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA88BF: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FBE7FF: cudaSetDevice (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248== 
==22248== 152 bytes in 1 blocks are possibly lost in loss record 1,337 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x36A52F94: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x368DD1B7: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA4CB4: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA88BF: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FBE7FF: cudaSetDevice (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248== 
==22248== 400 bytes in 1 blocks are possibly lost in loss record 1,462 of 1,502
==22248==    at 0x4843FD3: calloc (vg_replace_malloc.c:1595)
==22248==    by 0x40149CA: allocate_dtv (dl-tls.c:286)
==22248==    by 0x40149CA: _dl_allocate_tls (dl-tls.c:532)
==22248==    by 0x5788322: allocate_stack (allocatestack.c:622)
==22248==    by 0x5788322: pthread_create@@GLIBC_2.2.5 (pthread_create.c:660)
==22248==    by 0x36A119B5: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A51086: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36BA4F5D: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x36A0788F: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.470.223.02)
==22248==    by 0x5FA2D6E: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5FA4E70: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x579047E: __pthread_once_slow (pthread_once.c:116)
==22248==    by 0x5FE5AF8: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248==    by 0x5F9F58F: ??? (in /datav/lean/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.2.152)
==22248== 
==22248== LEAK SUMMARY:
==22248==    definitely lost: 0 bytes in 0 blocks
==22248==    indirectly lost: 0 bytes in 0 blocks
==22248==      possibly lost: 2,216 bytes in 14 blocks
==22248==    still reachable: 1,853,376 bytes in 4,630 blocks
==22248==                       of which reachable via heuristic:
==22248==                         stdstring          : 77,859 bytes in 655 blocks
==22248==         suppressed: 32 bytes in 1 blocks
==22248== Reachable blocks (those to which a pointer was found) are not shown.
==22248== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==22248== 
==22248== For lists of detected and suppressed errors, rerun with: -s
==22248== ERROR SUMMARY: 14 errors from 14 contexts (suppressed: 0 from 0)

2. C++查看内存

c++中能否判断一个指针指向栈还是堆？ - 知乎

函数调用栈区

  • main函数有一个栈区
    • main调用func1会有一个栈区；同时：push ebp ; ebp=esp; esp=esp-数值，这个数值由编译器决定
    • func栈区就在main栈区上面
    • 函数调用结束后，清理栈区，回到原来的main栈区
  • main栈区和func栈区到底多大，是由编译器决定的！！
    • 然后再func栈区里面申请临时变量，malloc的地址在堆上，然后把堆的地址给了临时变量；通过指针，把地址的内容传递出来。

3. 通过举例看下内存泄漏的问题

- valgrind检测的5种内存泄漏情况详解 valgrind检测的5种内存泄漏情况详解_still reachable-CSDN博客

3.1 malloc申请的内存没有释放——肯still reachable

debug可以发现：栈指针指向的是堆内存。

3.2 possibly lost

3.3 definitely lost

3.4 possibly lost

3.5 lost

 4. cuda demo测试——stream流句柄无法释放（cuda在主线程上）


// CUDA运行时头文件
#include <cuda_runtime.h>

#include <stdio.h>
#include <string.h>

#define checkRuntime(op)  __check_cuda_runtime((op), #op, __FILE__, __LINE__)

bool __check_cuda_runtime(cudaError_t code, const char* op, const char* file, int line){
    if(code != cudaSuccess){    
        const char* err_name = cudaGetErrorName(code);    
        const char* err_message = cudaGetErrorString(code);  
        printf("runtime error %s:%d  %s failed. \n  code = %s, message = %s\n", file, line, op, err_name, err_message);   
        return false;
    }
    return true;
}

int main(){
    char count = 0;
    do{
        int device_id = 0;
        checkRuntime(cudaSetDevice(device_id)); // CPU:1M->9M

        cudaStream_t stream = nullptr;
        // 这行语句执行后：显存开始有线程，并占用79M；内存从9M上升到100M
        checkRuntime(cudaStreamCreate(&stream));        // 创建一个stream

        // 在GPU上开辟空间
        float* memory_device = nullptr;
        checkRuntime(cudaMalloc(&memory_device, 100 * sizeof(float)));

        // 在CPU上开辟空间并且放数据进去，将数据复制到GPU
        float* memory_host = new float[100];
        memory_host[2] = 520.25; // Async就是异步操作，之前的都是 同步操作；异步复制时，发出指令立即返回，并不等待复制结束
        checkRuntime(cudaMemcpyAsync(memory_device, memory_host, sizeof(float) * 100, cudaMemcpyHostToDevice, stream)); // 异步复制操作，主线程不需要等待复制结束才继续

        // 在CPU上开辟pin memory,并将GPU上的数据复制回来 
        float* memory_page_locked = nullptr;
        checkRuntime(cudaMallocHost(&memory_page_locked, 100 * sizeof(float))); // 同样不等待复制完成，但在流中排队
        checkRuntime(cudaMemcpyAsync(memory_page_locked, memory_device, sizeof(float) * 100, cudaMemcpyDeviceToHost, stream)); // 异步复制操作，主线程不需要等待复制结束才继续
        
        printf("%f\n", memory_page_locked[2]); //    可能是错的，因为复制还没结束

        checkRuntime(cudaStreamSynchronize(stream));    // 统一等待流队列中的所有操作结束
        
        printf("%f\n", memory_page_locked[2]);
        
        // 释放内存
        checkRuntime(cudaFreeHost(memory_page_locked));
        checkRuntime(cudaFree(memory_device));
        checkRuntime(cudaStreamDestroy(stream));
        delete [] memory_host;

        count += 1;
    }while(count < 2);

    // 显存79M，内存103M；回不到原来去，这是没办法的
    return 0;
}

5. cuda在子线程上——资源仍然不会释放

关于线程：

  • 主线程太快了，子线程还没跑完，就丢了子线程；所以子线程的东西就没有打印了。 

_helen_520
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
浅谈python 导入模块和解决文件句柄找不到问题
12-24
如果你退出 Python 解释器并重新进入，你做的任何定义（变量和方法）都会丢失。因此，如果你想要编写一些更大的程序，为准备解释器输入使用一个文本编辑器会更好，并以那个文件替代作为输入执行。这就是传说中的脚本 Python 提供了一个方法可以从文件中获取定义，在脚本或者解释器的一个交互式实例中使用。这样的文件被称为模块。 导入模块： python导入模块默认是从sys.path的路径中查找。所以应该把这个模块放在sys.path的值对应的文件夹里。否则就找不到要导入的模块。如果在cmd中或者IDLE中运行python的话，每次修改过文件夹后，应该重新打开cmd或者IDLE，否则修改不生效。 运
解决显存占用/泄露，显示异常
AI炼丹师
06-15 958
#ps x |grep python|awk ‘{print $1}’|xargs kill 杀死所有僵尸进程 #root用户 fuser -v /dev/nvidia* |awk ‘{for(i=1;i
记录关于GPU显存泄露的问题
weixin_45120003的博客
11-06 122
RuntimeError: CUDA out of memory.
linux GPU显存泄漏解决方法
Learning from the mistakes
10-22 1646
今天发现gpu并没有进程，但是占用着显存，目测可能内存泄露了 我这里分享一下我的解决过程，我是root用户： apt install psmisc fuser -v /dev/nvidia* 可以发现很多僵尸进程： 然后全部杀死： lsof /dev/nvidia* | awk '{print $2}' | xargs -I {} kill {} 参考文献 GPU显存泄漏解决方法 linux 安装 fuser ...
如何查找DirectX3D应用程序的显存泄露
rabbit729的专栏
04-09 2983
在关闭一个基于D3D的应用程序时，D3D调试运行库会报告内存泄露。按照以下步骤定位泄漏点。1. 在DirectX Control Panel中（通常在DXSDK安装目录中可以找到），启用“Use Debug Version of Direct3D 9”并且将Debug Output Level设置为”More”。确保Break on Memory Leaks被禁用。点击Apply。2. 在VS中调
GPU显存泄露/显存溢出/显存爆炸 解决方案
dream6985的博客
08-07 2070
GPU显存泄露/显存溢出/显存爆炸 解决方案
valgrind检查still reachable情况
weixin_33856370的博客
06-22 6205
valgrind --leak-check=yes检查bufr编解码程序运行时提示still reachable: 568 bytes in 1 blocks，如下图示： 于是怀疑有内存泄漏，难道是malloc或calloc后没有free?但这样仍看不出哪里代码出了问题，于是开启valgrind的选项--show-reachable=yes再运行一遍，如下图示： 从上图可以看出问题代码...
valgrind检测的5种内存泄漏情况详解
qq_37535749的博客
04-30 5363
摘要： valgrind是linux下用于调试程序和查找内存泄露的常用工具。valgrind会报告5种内存泄露，“definitely lost”, “indirectly lost”, “possibly lost”, “still reachable”, and “suppressed”。笔者于工作闲暇之余对这5种（其实是4种，有一种没研究出结果）内存泄露的出现原因及区别进行了研究，撰此文以记之。 官方解释及分析： 摘自http://valgrind.org/docs/manual/faq.html#f
【转】解决内存/显存泄露的方法 pytorch
fanyue1997的博客
12-12 1552
def debug_memory(): import collections, gc, resource, torch print('maxrss = {}'.format( resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)) tensors = collections.Counter((str(o.device), o.dtype, tuple(o.shape)) .
解决神经网络训练时，代码没正常结束导致内存泄露显存不足的问题
欢迎一起成长呀
11-30 238
解决神经网络训练时，代码没正常结束导致内存泄露显存不足的问题
MySQL 句柄数占用过多的解决方法
09-10
本文主要介绍解决MySQL句柄占用过多的方法，简单实用，需要的朋友可以参考下。
解决C# 绑定句柄无效错误.doc
06-07
解决 C# 绑定句柄无效错误 错误代码1058错误 内有截图
Tensor是如何让你的内存/显存泄漏的
whaosoft143ai的博客
07-13 1103
某codebase出现了奇怪的泄漏现象，奇怪的点有以下几个方面：（1）不同的模型，内存/显存泄漏的现象不一样。比如A模型和B模型泄露的速度是不一样的（2）训练同一个模型的时候，如果在dataset中增加了数据量，相比不加数据，会在更早的epoch就把内存泄漏完。是不是听起来现象非常离谱，本着”code never lies“的世界观，我开始探求这个现象的真正原因。要想解决一个大的问题，首先就要降低问题的复杂度。最小复现代码是我们找问题的基础，而这个写最小复现代码的过程其实也是遵循了一定套路的，此处一并分享给
关于GPU资源泄露及解决办法
weixin_43380510的博客
01-10 2265
当在服务器上的GPU运行程序时，想通过Ctrl+C打断程序，但发现结束程序后GPU资源仍然被自己的进程占，此时需要kill掉你之前打断程序的进程，因此需要在命令行输入： ps 这时则会出现目前仍然在占用的进程号，然后就可以将此号kill掉，需要在命令行输入： kill -9 进程号 ...
关于SetTexture()调用引起显存泄露的问题
rabbit729的专栏
02-26 5623
项目中遇到一个问题，添加了GPU特技后，播放多次就会出现申请显存失败的问题，并且使用GPU-Z查看显存，也是在不断增加，于是断定是由于显存泄露造成的，仔细检查代码，发现在调用LPDIRECT3DTEXTURE9的Release时，返回值为1而不是0，这样的话实际上DX并没有真正释放显存，而只是将其引用计数减一，结果造成了显存泄露。问题在于DX的文档中只是说明了在调用LPDIRECT3DTEXTUR
valgrind中still reachable的解释
yk_home的博客
05-23 1841
stackoverflow上关于still reachable的回答： https://stackoverflow.com/questions/3840582/still-reachable-leak-detected-by-valgrind There is more than one way to define “memory leak”. In particular, there are two primary definitions of “memory leak” that are in co
查看显存使用情况：nvidia-smi
weixin_38620434的博客
03-27 3749
Pwr:Usager/Cap：能耗，Usage：使用量，Cap：总量，图中能耗使用量为320W/370W。GPU：GPU编号，多块显卡会从0开始编号，图中有两块GPU，一块GPU编号为 0，另一块编号为1。Perf：GPU性能状态，P0为最大性能，P12为最小性能，图中GPU性能状态为P2。Name：GPU类型，图中GPU类型为 NVIDIA GeForce 3090。Volatile GPU-Util：GPU使用率，图中GPU使用率为 92%Temp：GPU温度，图中GPU温度为 82℃。
Flask 推理MXnet 模型时显存泄漏
weixin_41683534的博客
01-19 1571
Flask、MXnet模型推理、显存泄漏
注意力机制略解
最新发布
cqbzcsq的博客
04-29 1000
本文简要地介绍了注意力机制，Q、K、V矩阵的含义，注意力评分函数的计算方式，并提及了Transformer的相关内容
怎么解决linux失效文件句柄
07-12
5. 更新或修复应用程序：如果失效的文件句柄是由特定应用程序引起的，可以尝试更新或修复该应用程序，以解决句柄失效问题。 请注意，在执行任何更改之前，请确保对系统进行适当的备份，并谨慎操作以避免对系统造成...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值