【Mindspore】使用Mindspore训练Faster RCNN报错：cause:TBEException:ERROR:

问题：

【功能模块】

系统：EulerOS  x86_64

mindspore-ascend 1.1.1.20210201

Package          Version

---------------- -------------------

addict           2.4.0

asttokens        2.0.4

astunparse       1.6.3

attrs            20.3.0

certifi          2020.12.5

cffi             1.14.5

chardet          4.0.0

cycler           0.10.0

Cython           0.29.22

decorator        4.4.2

easydict         1.9

hccl             0.1.0

idna             2.10

kiwisolver       1.3.1

matplotlib       3.3.4

mindspore-ascend 1.1.1.20210201

mmcv             0.2.14

mpmath           1.2.1

numpy            1.17.5

opencv-python    4.5.1.48

packaging        20.9

Pillow           8.1.2

pip              21.0.1

protobuf         3.15.5

psutil           5.8.0

pycocotools      2.0.2

pycparser        2.20

pyparsing        2.4.7

python-dateutil  2.8.1

PyYAML           5.4.1

requests         2.25.1

scipy            1.6.1

setuptools       52.0.0.post20210125

six              1.15.0

sympy            1.7.1

te               0.4.0

topi             0.4.0

urllib3          1.26.3

wheel            0.36.2

【操作步骤&问题现象】

1、从mindspore: MindSpore is a new open source deep learning training/inference framework that could be used for mobile, edge and cloud scenarios. 下载Faster RCNN源码

2、在src/config.py修改数据集路径（除此之外，未修改其它位置的源码）

3、使用src/convert_checkpoint.py脚本转换resnet预训练模型

4、使用scripts文件夹中的脚本训练：sh run_standalone_train_ascend.sh ../src/resnet50_backbone.ckpt

5、scripts文件夹生成train文件夹，train文件夹中log日志报错如下：

【截图信息】

[WARNING] ME(32429:140219627501376,MainProcess):2021-03-16-17:01:07.330.962 [mindspore/ops/operations/array_ops.py:2302] WARN_DEPRECATED: The usage of Pack is deprecated. Please use Stack.

[WARNING] ME(32429:140219627501376,MainProcess):2021-03-16-17:01:12.625.016 [mindspore/train/serialization.py:386] 30 parameters in the net are not loaded.

[WARNING] ME(33555:139667936442176,MainProcess):2021-03-16-17:03:08.886.486 [mindspore/ops/operations/array_ops.py:2302] WARN_DEPRECATED: The usage of Pack is deprecated. Please use Stack.

[WARNING] ME(33607:140071613749056,MainProcess):2021-03-16-17:03:15.350.267 [mindspore/ops/operations/array_ops.py:2302] WARN_DEPRECATED: The usage of Pack is deprecated. Please use Stack.

[WARNING] DEVICE(32429,python):2021-03-16-17:03:23.200.119 [mindspore/ccsrc/runtime/device/ascend/kernel_select_ascend.cc:482] SelectKernelInfo] The node [kernel_graph_1:[CNode]0{[0]: ValueNode<Primitive> TopK, [1]: [CNode]1, [2]: ValueNode<Tensor> Tensor(shape=[], dtype=Int64, value= 2000)}] cannot find valid TBE kernel info, try to get aicpu kernel info

[WARNING] DEVICE(32429,python):2021-03-16-17:03:23.200.358 [mindspore/ccsrc/runtime/device/ascend/kernel_select_ascend.cc:282] TagRaiseReduce] node:[TopK]reduce precision from int64 to int32

[WARNING] DEVICE(32429,python):2021-03-16-17:03:23.200.416 [mindspore/ccsrc/runtime/device/ascend/kernel_select_ascend.cc:282] TagRaiseReduce] node:[TopK]reduce precision from int64 to int32

[WARNING] DEVICE(32429,python):2021-03-16-17:03:23.342.200 [mindspore/ccsrc/runtime/device/ascend/kernel_select_ascend.cc:482] SelectKernelInfo] The node [kernel_graph_1:[CNode]0{[0]: ValueNode<Primitive> TopK, [1]: [CNode]1, [2]: ValueNode<Tensor> Tensor(shape=[], dtype=Int64, value= 2000)}] cannot find valid TBE kernel info, try to get aicpu kernel info

[WARNING] DEVICE(32429,python):2021-03-16-17:03:23.342.344 [mindspore/ccsrc/runtime/device/ascend/kernel_select_ascend.cc:282] TagRaiseReduce] node:[TopK]reduce precision from int64 to int32

[WARNING] DEVICE(32429,python):2021-03-16-17:03:23.342.390 [mindspore/ccsrc/runtime/device/ascend/kernel_select_ascend.cc:282] TagRaiseReduce] node:[TopK]reduce precision from int64 to int32

[WARNING] SESSION(32429,python):2021-03-16-17:03:28.628.333 [mindspore/ccsrc/backend/session/ascend_session.cc:1412] SelectKernel] There are 246 node/nodes used reduce precision to selected the kernel!

WARNING: 'ControlDepend' is deprecated from version 1.1 and will be removed in a future version, use 'Depend' instead.

[ERROR] KERNEL(32429,python):2021-03-16-17:08:05.756.789 [mindspore/ccsrc/backend/kernel_compiler/tbe/tbe_kernel_parallel_build.cc:88] TbeOpParallelBuild] task compile Failed, task id:984, cause:TBEException:ERROR:

Traceback (most recent call last):

  File "/disk1/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/_extends/parallel_compile/tbe_compiler/compiler.py", line 113, in build_op

    return op_func(*inputs_args, *outputs_args, *attrs_args, kernel_name=kernel_name)

  File "/root/miniconda3/envs/gwc/lib/python3.7/site-packages/te/utils/para_check.py", line 529, in _in_wrapper

    return func(*args, **kwargs)

  File "/usr/local/Ascend/opp/op_impl/built-in/ai_core/tbe/impl/select.py", line 305, in select

    "x1", error_detail)

  File "/root/miniconda3/envs/gwc/lib/python3.7/site-packages/te/utils/error_manager/error_manager_vector.py", line 229, in raise_err_two_input_shape_invalid

    raise RuntimeError(args_dict, msg)

RuntimeError: ({'errCode': 'E80029', 'op_name': 'Select_16037015126926824515_0', 'param_name1': 'condition', 'param_name2': 'x1', 'error_detail': 'Shape of tensor condition and x1 must be equal!'}, 'In op[Select_16037015126926824515_0], the shape of inputs[condition][x1] are invalid, [Shape of tensor condition and x1 must be equal!].')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/disk1/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/_extends/parallel_compile/tbe_compiler/compiler.py", line 154, in <module>

    result = compile_with_json(in_args)

  File "/disk1/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/_extends/parallel_compile/tbe_compiler/compiler.py", line 149, in compile_with_json

    ret = build_op(op_build, json_str)

  File "/disk1/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/_extends/parallel_compile/tbe_compiler/compiler.py", line 116, in build_op

    raise RuntimeError(e)

RuntimeError: ({'errCode': 'E80029', 'op_name': 'Select_16037015126926824515_0', 'param_name1': 'condition', 'param_name2': 'x1', 'error_detail': 'Shape of tensor condition and x1 must be equal!'}, 'In op[Select_16037015126926824515_0], the shape of inputs[condition][x1] are invalid, [Shape of tensor condition and x1 must be equal!].')

input_args: {"full_name":"Default/network-TrainOneStepCell/network-WithLossCell/_backbone-Faster_Rcnn_Resnet50/roi_align-SingleRoIExtractor/Select-op3058","gen_model":"single","impl_path":"","op_info":{"attrs":null,"inputs":[[{"dtype":"int8","format":"NC1HWC0","name":"condition_0","ori_format":"NCHW","ori_shape":[1280,256,7,7],"param_type":"required","range":[[1280,1280],[256,256],[7,7],[7,7]],"shape":[1280,8,7,7,32],"valid":true}],[{"dtype":"float32","format":"NC1HWC0","name":"x1_0","ori_format":"NCHW","ori_shape":[1280,256,7,7],"param_type":"required","range":[[1280,1280],[256,256],[7,7],[7,7]],"shape":[1280,16,7,7,16],"valid":true}],[{"dtype":"float32","format":"NC1HWC0","name":"x2_0","ori_format":"NCHW","ori_shape":[1280,256,7,7],"param_type":"required","range":[[1280,1280],[256,256],[7,7],[7,7]],"shape":[1280,16,7,7,16],"valid":true}]],"is_dynamic_shape":false,"kernel_name":"Select_16037015126926824515_0","name":"select","outputs":[[{"dtype":"float32","format":"NC1HWC0","name":"y","ori_format":"NCHW","ori_shape":[1280,256,7,7],"param_type":"required","range":[[1280,1280],[256,256],[7,7],[7,7]],"shape":[1280,16,7,7,16],"valid":true}]],"socVersion":"Ascend910A"},"platform":"TBE"} trace: 

In file /disk2/gwc/faster_rcnn0315/scripts/train/src/FasterRcnn/roi_align.py(177)/            res = self.select(mask, roi_feats_t, res)/

In file /disk2/gwc/faster_rcnn0315/scripts/train/src/FasterRcnn/faster_rcnn_r50.py(276)/            roi_feats = self.roi_align(rois,/

In file /disk2/gwc/faster_rcnn0315/scripts/train/src/network_define.py(181)/        loss1, loss2, loss3, loss4, loss5, loss6 = self.backbone(x, img_shape, gt_bboxe, gt_label, gt_num)/

In file /root/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/train/dataset_helper.py(87)/            return self.network(*outputs)/

WARNING: 'ControlDepend' is deprecated from version 1.1 and will be removed in a future version, use 'Depend' instead.

Start create dataset!

CHECKING MINDRECORD FILES ...

CHECKING MINDRECORD FILES DONE!

Create dataset done!

Traceback (most recent call last):

  File "train.py", line 178, in <module>

    model.train(config.epoch_size, dataset, callbacks=cb)

  File "/root/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/train/model.py", line 592, in train

    sink_size=sink_size)

  File "/root/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/train/model.py", line 391, in _train

    self._train_dataset_sink_process(epoch, train_dataset, list_callback, cb_params, sink_size)

  File "/root/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/train/model.py", line 452, in _train_dataset_sink_process

    outputs = self._train_network(*inputs)

  File "/root/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/nn/cell.py", line 322, in __call__

    out = self.compile_and_run(*inputs)

  File "/root/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/nn/cell.py", line 578, in compile_and_run

    self.compile(*inputs)

  File "/root/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/nn/cell.py", line 565, in compile

    _executor.compile(self, *inputs, phase=self.phase, auto_parallel_mode=self._auto_parallel_mode)

  File "/root/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/common/api.py", line 505, in compile

    result = self._executor.compile(obj, args_list, phase, use_vm)

RuntimeError: mindspore/ccsrc/backend/kernel_compiler/tbe/tbe_kernel_parallel_build.cc:88 TbeOpParallelBuild] ArgumentError task compile Failed, task id:984, cause:TBEException:ERROR:

Traceback (most recent call last):

  File "/disk1/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/_extends/parallel_compile/tbe_compiler/compiler.py", line 113, in build_op

    return op_func(*inputs_args, *outputs_args, *attrs_args, kernel_name=kernel_name)

  File "/root/miniconda3/envs/gwc/lib/python3.7/site-packages/te/utils/para_check.py", line 529, in _in_wrapper

    return func(*args, **kwargs)

  File "/usr/local/Ascend/opp/op_impl/built-in/ai_core/tbe/impl/select.py", line 305, in select

    "x1", error_detail)

  File "/root/miniconda3/envs/gwc/lib/python3.7/site-packages/te/utils/error_manager/error_manager_vector.py", line 229, in raise_err_two_input_shape_invalid

    raise RuntimeError(args_dict, msg)

RuntimeError: ({'errCode': 'E80029', 'op_name': 'Select_16037015126926824515_0', 'param_name1': 'condition', 'param_name2': 'x1', 'error_detail': 'Shape of tensor condition and x1 must be equal!'}, 'In op[Select_16037015126926824515_0], the shape of inputs[condition][x1] are invalid, [Shape of tensor condition and x1 must be equal!].')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/disk1/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/_extends/parallel_compile/tbe_compiler/compiler.py", line 154, in <module>

    result = compile_with_json(in_args)

  File "/disk1/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/_extends/parallel_compile/tbe_compiler/compiler.py", line 149, in compile_with_json

    ret = build_op(op_build, json_str)

  File "/disk1/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/_extends/parallel_compile/tbe_compiler/compiler.py", line 116, in build_op

    raise RuntimeError(e)

RuntimeError: ({'errCode': 'E80029', 'op_name': 'Select_16037015126926824515_0', 'param_name1': 'condition', 'param_name2': 'x1', 'error_detail': 'Shape of tensor condition and x1 must be equal!'}, 'In op[Select_16037015126926824515_0], the shape of inputs[condition][x1] are invalid, [Shape of tensor condition and x1 must be equal!].')

input_args: {"full_name":"Default/network-TrainOneStepCell/network-WithLossCell/_backbone-Faster_Rcnn_Resnet50/roi_align-SingleRoIExtractor/Select-op3058","gen_model":"single","impl_path":"","op_info":{"attrs":null,"inputs":[[{"dtype":"int8","format":"NC1HWC0","name":"condition_0","ori_format":"NCHW","ori_shape":[1280,256,7,7],"param_type":"required","range":[[1280,1280],[256,256],[7,7],[7,7]],"shape":[1280,8,7,7,32],"valid":true}],[{"dtype":"float32","format":"NC1HWC0","name":"x1_0","ori_format":"NCHW","ori_shape":[1280,256,7,7],"param_type":"required","range":[[1280,1280],[256,256],[7,7],[7,7]],"shape":[1280,16,7,7,16],"valid":true}],[{"dtype":"float32","format":"NC1HWC0","name":"x2_0","ori_format":"NCHW","ori_shape":[1280,256,7,7],"param_type":"required","range":[[1280,1280],[256,256],[7,7],[7,7]],"shape":[1280,16,7,7,16],"valid":true}]],"is_dynamic_shape":false,"kernel_name":"Select_16037015126926824515_0","name":"select","outputs":[[{"dtype":"float32","format":"NC1HWC0","name":"y","ori_format":"NCHW","ori_shape":[1280,256,7,7],"param_type":"required","range":[[1280,1280],[256,256],[7,7],[7,7]],"shape":[1280,16,7,7,16],"valid":true}]],"socVersion":"Ascend910A"},"platform":"TBE"} trace: 

In file /disk2/gwc/faster_rcnn0315/scripts/train/src/FasterRcnn/roi_align.py(177)/            res = self.select(mask, roi_feats_t, res)/

In file /disk2/gwc/faster_rcnn0315/scripts/train/src/FasterRcnn/faster_rcnn_r50.py(276)/            roi_feats = self.roi_align(rois,/

In file /disk2/gwc/faster_rcnn0315/scripts/train/src/network_define.py(181)/        loss1, loss2, loss3, loss4, loss5, loss6 = self.backbone(x, img_shape, gt_bboxe, gt_label, gt_num)/

In file /root/miniconda3/envs/gwc/lib/python3.7/site-packages/mindspore/train/dataset_helper.py(87)/            return self.network(*outputs)/

解决方案：

尝试重装te和topi

pip uninstall te -y
pip uninstall topi -y
pip install /usr/local/Ascend/fwkacllib/lib64/topi-*.whl
pip install /usr/local/Ascend/fwkacllib/lib64/te-*.whl

Atlas的包的话

pip uninstall te topi hccl -y
pip install /usr/local/Ascend/ascend-toolkit/latest/fwkacllib/lib64/topi-{version}-py3-none-any.whl
pip install /usr/local/Ascend/ascend-toolkit/latest/fwkacllib/lib64/te-{version}-py3-none-any.whl
pip install /usr/local/Ascend/ascend-toolkit/latest/fwkacllib/lib64/hccl-{version-py3-none-any.whl