1、问题描述
在GPU环境下运行dlib中的例子dnn_mmod_ex时,报错:
terminate called after throwing an instance of 'dlib::cuda_error'
what(): Error while calling cudaMalloc(&data, new_size*sizeof(float)) in file /home/laoer/tools/dlib/dlib-19.17/dlib/cuda/gpu_data.cpp:218. code: 2, reason: out of memory
The program has unexpectedly finished.
2、原因分析
报错信息中“gpu_data.cpp:218. code: 2, reason: out of memory”,GPU内存不足,查看代码,在加载小批量数据时有:
cropper(150, images_train, face_boxes_train, mini_batch_samples, mini_batch_labels);
说明在训练时,加载的小批量数据太多,GPU内存没法装下这么多数据。
3、解决方法
减小加载小批量的数据量,如150改为50
cropper(50, images_train, face_boxes_train, mini_batch_samples, mini_batch_labels);
理论上改小后,模型训练时间会变长,但是精度会提高。