FastMaskRCNN运行错误排查

其git地址为：https://github.com/CharlesShang/FastMaskRCNN，按照其步骤描述进行环境配置，在最后一步运行的时候，出现

Restored 267(640) vars from ./data/pretrained_models/resnet_v1_50.ckpt
2017-12-03 20:46:12.688523: W tensorflow/core/framework/op_kernel.cc:1192] Internal: WhereOp: Could not launch cub::DeviceReduce::Sum to count number of true indices.  temp_storage_bytes: 1, status: invalid device function
2017-12-03 20:46:12.721948: W tensorflow/core/framework/op_kernel.cc:1192] Internal: WhereOp: Could not launch cub::DeviceReduce::Sum to count number of true indices.  temp_storage_bytes: 1, status: invalid device function
     [[Node: pyramid_1/AssignGTBoxes/Where_5 = Where[_device="/job:localhost/replica:0/task:0/device:GPU:0"](pyramid_1/AssignGTBoxes/Equal_5/_1123)]]
2017-12-03 20:46:12.722069: W tensorflow/core/framework/op_kernel.cc:1192] Internal: WhereOp: Could not launch cub::DeviceReduce::Sum to count number of true indices.  temp_storage_bytes: 1, status: invalid device function
     [[Node: pyramid_1/AssignGTBoxes/Where_5 = Where[_device="/job:localhost/replica:0/task:0/device:GPU:0"](pyramid_1/AssignGTBoxes/Equal_5/_1123)]]
2017-12-03 20:46:12.722069: W tensorflow/core/framework/op_kernel.cc:1192] Internal: WhereOp: Could not launch cub::DeviceReduce::Sum to count number of true indices.  temp_storage_bytes: 1, status: invalid device function
     [[Node: pyramid_1/AssignGTBoxes/Where_5 = Where[_device="/job:localhost/replica:0/task:0/device:GPU:0"](pyramid_1/AssignGTBoxes/Equal_5/_1123)]]
Traceback (most recent call last):
  File "train/train.py", line 340, in <module>
    train()
  File "train/train.py", line 272, in train
    [input_image] + [final_box] + [final_cls] + [final_prob] + [final_gt_cls] + [gt] + [tmp_0] + [tmp_1] + [tmp_2] + [tmp_3] + [tmp_4])
  File "/home/machine/anaconda/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/home/machine/anaconda/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/machine/anaconda/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
    options, run_metadata)
  File "/home/machine/anaconda/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: WhereOp: Could not launch cub::DeviceReduce::Sum to count number of true indices.  temp_storage_bytes: 1, status: invalid device function
     [[Node: pyramid_1/AssignGTBoxes/Where_5 = Where[_device="/job:localhost/replica:0/task:0/device:GPU:0"](pyramid_1/AssignGTBoxes/Equal_5/_1123)]]
     [[Node: pyramid_2/OneHotEncoding_4/one_hot/_1327 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_9981_pyramid_2/OneHotEncoding_4/one_hot", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op u'pyramid_1/AssignGTBoxes/Where_5', defined at:
  File "train/train.py", line 340, in <module>
    train()
  File "train/train.py", line 193, in train
    loss_weights=[0.2, 0.2, 1.0, 0.2, 1.0])
  File "train/../libs/nets/pyramid_network.py", line 580, in build
    is_training=is_training, gt_boxes=gt_boxes)
  File "train/../libs/nets/pyramid_network.py", line 263, in build_heads
    assign_boxes(rois, [rois, batch_inds], [2, 3, 4, 5])
  File "train/../libs/layers/wrapper.py", line 172, in assign_boxes
    inds = tf.where(tf.equal(assigned_layers, l))
  File "/home/machine/anaconda/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 2439, in where
    return gen_array_ops.where(input=condition, name=name)
  File "/home/machine/anaconda/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 5930, in where
    "Where", input=input, name=name)
  File "/home/machine/anaconda/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/machine/anaconda/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
    op_def=op_def)
  File "/home/machine/anaconda/envs/tensorflow2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InternalError (see above for traceback): WhereOp: Could not launch cub::DeviceReduce::Sum to count number of true indices.  temp_storage_bytes: 1, status: invalid device function
     [[Node: pyramid_1/AssignGTBoxes/Where_5 = Where[_device="/job:localhost/replica:0/task:0/device:GPU:0"](pyramid_1/AssignGTBoxes/Equal_5/_1123)]]
     [[Node: pyramid_2/OneHotEncoding_4/one_hot/_1327 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_9981_pyramid_2/OneHotEncoding_4/one_hot", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

然后在各大网站上寻找解决方法，其实都没有找到。最后在https://github.com/CharlesShang/FastMaskRCNN/issues/159，给出了一个方法说是将tensorflow 1.3降到1.1。

我查看了一下我的tensorfow版本

>>> tf.__version__
'1.4.0'

果断重新新建一个python虚拟环境，安装tensorflow1.1，如下

conda create -n tensorflow-1.1 python=2.7
source activate tensorflow-1.1
wget tensorflow的安装地址
pip install tensorfow安装包

pip install opencv-python
conda install PIL
pip install scikit-image

然后重新运行就ok了。

Appendix

tensorflow-1.1的安装地址请从这里下载

https://pypi.python.org/pypi/tensorflow/1.1.0rc2 (cpu)
https://pypi.python.org/pypi/tensorflow-gpu/1.1.0rc2 (gpu)

注意python的版本。

---

除了以上下载安装tensorflow的方法，还可以使用如下代码安装

//先卸载
pip uninstall tensorflow-gpu
//再安装
pip install tensorflow-gpu==1.2
or
pip install tensorflow-gpu==1.3