*** Error in `../../build/tools/caffe.bin': malloc(): memory corruption (fast): 0x0000000006598790 ***
*** Aborted at 1498809502 (unix time) try "date -d @1498809502" if you are using GNU date ***
PC: @ 0x7f41a7159cc9 (unknown)
*** SIGABRT (@0x272100000ec6) received by PID 3782 (TID 0x7f41a8753780) from PID 3782; stack trace: ***
这个错误是在保存caffemodels的时候出错
F0630 15:01:50.368716 6670 syncedmem.cpp:51] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
GPU不够用,nvidia-smi命令可以查看GPU的使用情况,选择占有率的低,即可解决问题。或者使用多块GPU,也可以把batch_size调小,也能解决问题。
F0630 16:57:50.218991 19894 softmax_loss_layer.cpp:38] Check failed: this->layer_param_.loss_param().class_weighting_size() == bottom[0]->channels() (6 vs. 11) Number of class weight values does not match the number of classes.
(6 vs. 11)这种错误主要是不匹配,可能是层层之间的map不匹配;或者最后一层的out和类别不匹配;也有可能是caffemodels中只有权重和网络中的权重、偏置不匹配。