TypeError: ‘numpy.float64’ object cannot be interpreted as an index
解决:sudo pip install -U numpy==1.11.0
测试faster的时候出错:
../lib/datasets/voc_eval.py", line 148, in voc_eval
BB = BB[sorted_ind, :]
IndexError: too many indices for array
同类问题解决方法:http://blog.csdn.net/qq_26569761/article/details/52136845
和:http://blog.csdn.net/weinaonao5418/article/details/54234684
总的来说如果数据没问题的话,那就是说明模型没有训练好。
还有一种可能就是网络模型某些层没有初始化,即没有加载预训练模型,也没有随机初始化(高斯或者xavier)
修改数据集时出现错误:
I1022 15:50:30.147821 31371 solver.cpp:60] Solver scaffolding done.
Traceback (most recent call last):
File "./tools/train_net.py", line 112, in <module>
max_iters=args.max_iters)
File "/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 157, in train_net
pretrained_model=pretrained_model)
File "/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 53, in __init__
self.solver.net.layers[0].set_roidb(roidb)
File "/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/roi_data_layer/layer.py", line 68, in set_roidb
self._shuffle_roidb_inds()
File "/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/roi_data_layer/layer.py", line 35, in _shuffle_roidb_inds
inds = np.reshape(inds, (-1, 2))
File "/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 225, in reshape
return reshape(newshape, order=order)
ValueError: total size of new array must be unchanged
解决办法:
http://blog.csdn.net/zahuopu__xu/article/details/76438991
这里是 reshape方法报错, 确保tarin.txt, trainval.txt里的文件数量为偶数。则解决报错问题。
训练出错:删除(或减少dropout ratio)全链接的dropout层时出错,训练终止:
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:48: RuntimeWarning: overflow encountered in exp
pred_w = np.exp(dw) * widths[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:49: RuntimeWarning: overflow encountered in exp
pred_h = np.exp(dh) * heights[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:49: RuntimeWarning: overflow encountered in multiply
pred_h = np.exp(dh) * heights[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/rpn/proposal_layer.py:175: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
./experiments/scripts/faster_rcnn_end2end.sh: line 58: 4353 Floating point exception(core dumped) ./tools/train_net.py --gpu ${GPU_ID} --solver models/${PT_DIR}/${NET}/faster_rcnn_end2end/solver.prototxt --weights data/imagenet_models/zf_faster_rcnn_iter_50000.caffemodel --imdb ${TRAIN_IMDB} --iters ${ITERS} --cfg experiments/cfgs/faster_rcnn_end2end.yml ${EXTRA_ARGS}
测试时出错:
File "tools/test_net.py", line 90, in <module>
test_net(net, imdb, max_per_image=args.max_per_image, vis=args.vis)
File "/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/test.py", line 258, in test_net
scores, boxes = im_detect(net, im, box_proposals)
File "/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/test.py", line 172, in im_detect
box_deltas = blobs_out['bbox_predd'] #yuanlaishi bbox_pred
KeyError: 'bbox_predd'
由于修改了网络中bbox_predd的名字,需要在lib/fast_rcnn/test.py line 172 也修改为相同的名字
训练mobileNet时出错
I0105 11:18:01.993641 3280 net.cpp:106] Creating Layer conv5_5/dw
I0105 11:18:01.993644 3280 net.cpp:454] conv5_5/dw <- conv5_4/sep
I0105 11:18:01.993649 3280 net.cpp:411] conv5_5/dw -> conv5_5/dw
F0105 11:18:02.008042 3280 cudnn_conv_layer.cpp:53] Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0) CUDNN_STATUS_INTERNAL_ERROR
*** Check failure stack trace: ***
./experiments/scripts/faster_rcnn_end2end.sh: line 58: 3280 Aborted (core dumped) ./tools/train_net.py --gpu ${GPU_ID} --solver models/${PT_DIR}/${NET}/faster_rcnn_end2end/solver.prototxt --weights data/imagenet_models/mobilenet.caffemodel --imdb ${TRAIN_IMDB} --iters ${ITERS} --cfg experiments/cfgs/faster_rcnn_end2end.yml ${EXTRA_ARGS}
网络太大了,GPU显存不够
训练ShuffleNet时出错:
resx5_concat <- resx5_conv3
I0117 10:48:45.589972 4565 net.cpp:411] resx5_concat -> resx5_concat
F0117 10:48:45.589978 4565 concat_layer.cpp:42] Check failed: top_shape[j] == bottom[i]->shape(j) (26 vs. 27) All inputs must have the same shape, except at concat_axis.
*** Check failure stack trace: ***
这是由于Resx5层里面对特征图进行了缩放,
Pooling层当步长为2进行缩减一半时,是向上取整的;而depthwise层却是向下取整的;
这就导致当输入特征图为(53,64)时两者输出分别为(27,32)和(26,32)故无法进行concat操作。
所以只需在使用pooling层时对pad进行设置就好:
例如: pad_h: 1
pad_w: 0
此时: resx5_match_conv: Top shape: 1 240 27 32 (207360)
resx5_concat :Top shape: 1 480 27 32 (414720)
将CenterLoss调大以后报错:
I0226 15:37:05.994953 6019 solver.cpp:229] Iteration 0, loss = 6.69855
I0226 15:37:05.994982 6019 solver.cpp:245] Train net output #0: bbox_loss = 0.0246186 (* 1 = 0.0246186 loss)
I0226 15:37:05.994988 6019 solver.cpp:245] Train net output #1: center_loss = 87.7812 (* 0.005 = 0.438906 loss)
I0226 15:37:05.994993 6019 solver.cpp:245] Train net output #2: cls_loss = 3.64286 (* 1 = 3.64286 loss)
I0226 15:37:05.994997 6019 solver.cpp:245] Train net output #3: rpn_cls_loss = 1.25546 (* 1 = 1.25546 loss)
I0226 15:37:05.995002 6019 solver.cpp:245] Train net output #4: rpn_loss_bbox = 0.238083 (* 1 = 0.238083 loss)
I0226 15:37:05.995015 6019 sgd_solver.cpp:106] Iteration 0, lr = 0.001
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:48: RuntimeWarning: overflow encountered in exp
pred_w = np.exp(dw) * widths[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:48: RuntimeWarning: overflow encountered in multiply
pred_w = np.exp(dw) * widths[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:49: RuntimeWarning: overflow encountered in exp
pred_h = np.exp(dh) * heights[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:49: RuntimeWarning: overflow encountered in multiply
pred_h = np.exp(dh) * heights[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/rpn/proposal_layer.py:175: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
./experiments/scripts/faster_rcnn_end2end.sh: line 58: 6019 Floating point exception(core dumped) ./tools/train_net.py --gpu ${GPU_ID} --solver models/${PT_DIR}/${NET}/faster_rcnn_end2end/solver.prototxt --weights data/imagenet_models/ZF_faster_rcnn_final.caffemodel --imdb ${TRAIN_IMDB} --iters ${ITERS} --cfg experiments/cfgs/faster_rcnn_end2end.yml ${EXTRA_ARGS}
出现溢出;
解决方法:调小CenterLoss的权值,或降低学习率
解决:sudo pip install -U numpy==1.11.0
测试faster的时候出错:
../lib/datasets/voc_eval.py", line 148, in voc_eval
BB = BB[sorted_ind, :]
IndexError: too many indices for array
同类问题解决方法:http://blog.csdn.net/qq_26569761/article/details/52136845
和:http://blog.csdn.net/weinaonao5418/article/details/54234684
总的来说如果数据没问题的话,那就是说明模型没有训练好。
还有一种可能就是网络模型某些层没有初始化,即没有加载预训练模型,也没有随机初始化(高斯或者xavier)
修改数据集时出现错误:
I1022 15:50:30.147821 31371 solver.cpp:60] Solver scaffolding done.
Traceback (most recent call last):
File "./tools/train_net.py", line 112, in <module>
max_iters=args.max_iters)
File "/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 157, in train_net
pretrained_model=pretrained_model)
File "/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/train.py", line 53, in __init__
self.solver.net.layers[0].set_roidb(roidb)
File "/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/roi_data_layer/layer.py", line 68, in set_roidb
self._shuffle_roidb_inds()
File "/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/roi_data_layer/layer.py", line 35, in _shuffle_roidb_inds
inds = np.reshape(inds, (-1, 2))
File "/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 225, in reshape
return reshape(newshape, order=order)
ValueError: total size of new array must be unchanged
解决办法:
http://blog.csdn.net/zahuopu__xu/article/details/76438991
这里是 reshape方法报错, 确保tarin.txt, trainval.txt里的文件数量为偶数。则解决报错问题。
训练出错:删除(或减少dropout ratio)全链接的dropout层时出错,训练终止:
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:48: RuntimeWarning: overflow encountered in exp
pred_w = np.exp(dw) * widths[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:49: RuntimeWarning: overflow encountered in exp
pred_h = np.exp(dh) * heights[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:49: RuntimeWarning: overflow encountered in multiply
pred_h = np.exp(dh) * heights[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/rpn/proposal_layer.py:175: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
./experiments/scripts/faster_rcnn_end2end.sh: line 58: 4353 Floating point exception(core dumped) ./tools/train_net.py --gpu ${GPU_ID} --solver models/${PT_DIR}/${NET}/faster_rcnn_end2end/solver.prototxt --weights data/imagenet_models/zf_faster_rcnn_iter_50000.caffemodel --imdb ${TRAIN_IMDB} --iters ${ITERS} --cfg experiments/cfgs/faster_rcnn_end2end.yml ${EXTRA_ARGS}
测试时出错:
File "tools/test_net.py", line 90, in <module>
test_net(net, imdb, max_per_image=args.max_per_image, vis=args.vis)
File "/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/test.py", line 258, in test_net
scores, boxes = im_detect(net, im, box_proposals)
File "/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/test.py", line 172, in im_detect
box_deltas = blobs_out['bbox_predd'] #yuanlaishi bbox_pred
KeyError: 'bbox_predd'
由于修改了网络中bbox_predd的名字,需要在lib/fast_rcnn/test.py line 172 也修改为相同的名字
训练mobileNet时出错
I0105 11:18:01.993641 3280 net.cpp:106] Creating Layer conv5_5/dw
I0105 11:18:01.993644 3280 net.cpp:454] conv5_5/dw <- conv5_4/sep
I0105 11:18:01.993649 3280 net.cpp:411] conv5_5/dw -> conv5_5/dw
F0105 11:18:02.008042 3280 cudnn_conv_layer.cpp:53] Check failed: status == CUDNN_STATUS_SUCCESS (4 vs. 0) CUDNN_STATUS_INTERNAL_ERROR
*** Check failure stack trace: ***
./experiments/scripts/faster_rcnn_end2end.sh: line 58: 3280 Aborted (core dumped) ./tools/train_net.py --gpu ${GPU_ID} --solver models/${PT_DIR}/${NET}/faster_rcnn_end2end/solver.prototxt --weights data/imagenet_models/mobilenet.caffemodel --imdb ${TRAIN_IMDB} --iters ${ITERS} --cfg experiments/cfgs/faster_rcnn_end2end.yml ${EXTRA_ARGS}
网络太大了,GPU显存不够
训练ShuffleNet时出错:
resx5_concat <- resx5_conv3
I0117 10:48:45.589972 4565 net.cpp:411] resx5_concat -> resx5_concat
F0117 10:48:45.589978 4565 concat_layer.cpp:42] Check failed: top_shape[j] == bottom[i]->shape(j) (26 vs. 27) All inputs must have the same shape, except at concat_axis.
*** Check failure stack trace: ***
这是由于Resx5层里面对特征图进行了缩放,
Pooling层当步长为2进行缩减一半时,是向上取整的;而depthwise层却是向下取整的;
这就导致当输入特征图为(53,64)时两者输出分别为(27,32)和(26,32)故无法进行concat操作。
所以只需在使用pooling层时对pad进行设置就好:
例如: pad_h: 1
pad_w: 0
此时: resx5_match_conv: Top shape: 1 240 27 32 (207360)
resx5_concat :Top shape: 1 480 27 32 (414720)
将CenterLoss调大以后报错:
I0226 15:37:05.994953 6019 solver.cpp:229] Iteration 0, loss = 6.69855
I0226 15:37:05.994982 6019 solver.cpp:245] Train net output #0: bbox_loss = 0.0246186 (* 1 = 0.0246186 loss)
I0226 15:37:05.994988 6019 solver.cpp:245] Train net output #1: center_loss = 87.7812 (* 0.005 = 0.438906 loss)
I0226 15:37:05.994993 6019 solver.cpp:245] Train net output #2: cls_loss = 3.64286 (* 1 = 3.64286 loss)
I0226 15:37:05.994997 6019 solver.cpp:245] Train net output #3: rpn_cls_loss = 1.25546 (* 1 = 1.25546 loss)
I0226 15:37:05.995002 6019 solver.cpp:245] Train net output #4: rpn_loss_bbox = 0.238083 (* 1 = 0.238083 loss)
I0226 15:37:05.995015 6019 sgd_solver.cpp:106] Iteration 0, lr = 0.001
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:48: RuntimeWarning: overflow encountered in exp
pred_w = np.exp(dw) * widths[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:48: RuntimeWarning: overflow encountered in multiply
pred_w = np.exp(dw) * widths[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:49: RuntimeWarning: overflow encountered in exp
pred_h = np.exp(dh) * heights[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/fast_rcnn/bbox_transform.py:49: RuntimeWarning: overflow encountered in multiply
pred_h = np.exp(dh) * heights[:, np.newaxis]
/media/snowflake/12078d75-48b0-4714-b5ea-0a5fcd247ddc/py-faster-rcnn/tools/../lib/rpn/proposal_layer.py:175: RuntimeWarning: invalid value encountered in greater_equal
keep = np.where((ws >= min_size) & (hs >= min_size))[0]
./experiments/scripts/faster_rcnn_end2end.sh: line 58: 6019 Floating point exception(core dumped) ./tools/train_net.py --gpu ${GPU_ID} --solver models/${PT_DIR}/${NET}/faster_rcnn_end2end/solver.prototxt --weights data/imagenet_models/ZF_faster_rcnn_final.caffemodel --imdb ${TRAIN_IMDB} --iters ${ITERS} --cfg experiments/cfgs/faster_rcnn_end2end.yml ${EXTRA_ARGS}
出现溢出;
解决方法:调小CenterLoss的权值,或降低学习率