问题描述:
【功能模块】
mindspore
【操作步骤&问题现象】
1、用cifar10数据集训练网络
2、model.eval(dataset_eval)
在NPU上正常。
在GPU上:如果amp_level=O0,则正常返回准确率;如果amp_level=O2,就会打印如下报错信息:
[ERROR] DEVICE(28009,python3.7):2021-03-25-17:14:07.696.542 [mindspore/ccsrc/runtime/device/gpu/kernel_info_setter.cc:118] SelectAkgKernel] Not find op[BatchNorm] in akg
[ERROR] DEVICE(28009,python3.7):2021-03-25-17:14:07.696.594 [mindspore/ccsrc/runtime/device/gpu/kernel_info_setter.cc:322] PrintUnsupportedTypeException] Select GPU kernel op[BatchNorm] fail! Incom
patible data type!
The supported data types are in[float32 float32 float32 float32 float32], out[float32 float32 float32 float32 float32]; in[float16 float32 float32 float32 float32], out[float16 float32 float32 floa
t32 float32]; , but get in [float16 float16 float16 float16 float16 ] out [float16 float16 float16 float16 float16 ]
Traceback (most recent call last):
File "train.py", line 150, in <module>
acc = model.eval(ds_eval)
File "/opt/anaconda3/envs/python3.7/lib/python3.7/site-packages/mindspore/train/model.py", line 717, in eval
return self._eval_dataset_sink_process(valid_dataset, list_callback, cb_params)
File "/opt/anaconda3/envs/python3.7/lib/python3.7/site-packages/mindspore/train/model.py", line 621, in _eval_dataset_sink_process
outputs = self._eval_network(*inputs)
File "/opt/anaconda3/envs/python3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 322, in __call__
out = self.compile_and_run(*inputs)
File "/opt/anaconda3/envs/python3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 578, in compile_and_run
self.compile(*inputs)
File "/opt/anaconda3/envs/python3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 565, in compile
_executor.compile(self, *inputs, phase=self.phase, auto_parallel_mode=self._auto_parallel_mode)
File "/opt/anaconda3/envs/python3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 505, in compile
result = self._executor.compile(obj, args_list, phase, use_vm)
TypeError: mindspore/ccsrc/runtime/device/gpu/kernel_info_setter.cc:322 PrintUnsupportedTypeException] Select GPU kernel op[BatchNorm] fail! Incompatible data type!
The supported data types are in[float32 float32 float32 float32 float32], out[float32 float32 float32 float32 float32]; in[float16 float32 float32 float32 float32], out[float16 float32 float32 floa
t32 float32]; , but get in [float16 float16 float16 float16 float16 ] out [float16 float16 float16 float16 float16 ]
# In file /home/huawei/longtingting/ms-seng/seng_layer.py(593)
output = self.matmul_bt(x, self.weight)
【截图信息】
amp_level=O2时的报错信息:
【日志信息】(可选,上传日志内容或者附件)
解决方案:
GPU推理中的BatchNorm算子不支持纯fp16输入,推理应该不需要开启混合精度。 所以你推理时的model不要共用训练时候的。
一般有两种方式去得到精度结果:1 使用独立的eval过程,重新定义eval的net并对保存的ckpt进行推理。2 边训练边推理,这种方式需要使用回调函数进行验证。 这两种方式都可以参考master上,model_zoo/offical/cv/resnet/中的用法。