voc_eval.py:41: RuntimeWarning: invalid value encountered in greater

训练faster-rcnn报错

训练结束后测试时出现类似

File "/home/xxx/py-faster-rcnn/tools/../lib/datasets/voc_eval.py", line 126, in voc_eval
    R = [obj for obj in recs[imagename] if obj['name'] == classname]
KeyError: '000002'

解决方法: 删除data/VOCdekit2007下的annotations_cache文件夹

基于ResNet101训练py-faster-rcnn遇到错误：

/faster-rcnn-py/tools/../lib/fast_rcnn/bbox_transform.py:50: RuntimeWarning: overflow encountered in exp
pred_h = np.exp(dh) * heights[:, np.newaxis]
faster-rcnn-py/tools/../lib/rpn/proposal_layer.py:176: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]

类似解决方法：减小lr from 0.001 to 0.0001

A possible solution could be to decrease the base learning rate in the solver.prototxt
As it is recommended here http://caffe.berkeleyvision.org/tutorial/solver.html
Just change the base_lr: 0.001 to 0.0001

Note also that the above settings are merely guidelines, and they’re definitely not guaranteed to be optimal (or even work at all!) in every situation. If learning diverges (e.g., you start to see very large or NaN or inf loss values or outputs), try dropping the base_lr (e.g., base_lr: 0.001) and re-training, repeating this until you find a base_lr value that works.
I did try to change the base_lr value and now the NAN value disappeared.