源码下载:
https://gitee.com/open-ascend/atlas_mindxsdk_samples/blob/master/contrib/cv/object_detection/image_crnn_resnet50_yolov5
快速运行攻略(MindX SDK环境已经部署完毕情况下):
1、获取模型文件
(1)crnn.onnx文件
https://gitee.com/ai_samples/pytorch_models/tree/master/cv/ocr/crnn
存放到 image_crnn_resnet50_yolov5/data/models/crnn 目录下
(2)resnet50_pytorch_1.3.onnx文件
https://gitee.com/ai_samples/pytorch_models/tree/master/cv/classification/resnet50
存放到 image_crnn_resnet50_yolov5/data/models/resnet50 目录下
(3)yolov5s_sim_t.onnx文件
https://gitee.com/ai_samples/pytorch_models/tree/master/cv/object_detection/yolov5
存放到 image_crnn_resnet50_yolov5/data/models/yolov5 目录下
2、模型文件转换
(1)image_crnn_resnet50_yolov5/data/models/crnn目录下执行模型转换,根据芯片类型,执行atc_310.sh 或 atc_310P3.sh
(2)image_crnn_resnet50_yolov5/data/models/resnet50目录下执行模型转换,根据芯片类型,执行atc_310.sh 或 atc_310P3.sh
(3)image_crnn_resnet50_yolov5/data/models/yolov5目录下执行模型转换,根据芯片类型,执行atc_310.sh 或 atc_310P3.sh
bash atc_310.sh
bash atc_310P3.sh
3、修改run_cpp.sh & run_python.sh中MX_SDK_HOME为MindX SDK安装目录
export MX_SDK_HOME=/usr/local/sdk_home/mxVision
4、执行run_cpp.sh 或者 run_python.sh
bash run_cpp.sh
bash run_python.sh
一、安装昇腾驱动
先安装昇腾驱动,昇腾驱动请参考各个产品安装手册,安装完成后npu-smi info 显示安装成功
[root@localhost ~]#
[root@localhost ~]# npu-smi info
+-------------------------------------------------------------------------------------------------+
| npu-smi 22.0.2 Version: 22.0.2 |
+------------------+--------------+---------------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+==================+==============+===============================================================+
| 1 310 | OK | 12.8 45 0 / 0 |
| 0 0 | 0000:05:00.0 | 0 2621 / 8192 |
+==================+==============+===============================================================+
二、安装MindX SDK > mxVision
(1)MindX SDK需要通过官网获取。
(2)mxVision说明手册:
https://www.hiascend.com/document/detail/zh/mind-sdk/30rc3/quickstart/visionquickstart/visionquickstart_0000.html
(3)安装MindX SDK
./Ascend-mindxsdk-mxvision_3.0.RC2_linux-aarch64.run --install --install-path=/usr/local/sdk_home
–install-path为指定安装的路径
(4)安装成功后会提示如下信息
Installing collected packages:mindx
Successfully installed mindx-3.0.RC2
(5)安装成功后在对应目录下查看,能看到mxVision
[root@localhost sdk_home]#
[root@localhost sdk_home]# pwd
/usr/local/sdk_home
[root@localhost sdk_home]# ls
mxVision mxVision-3.0.RC2
[root@localhost sdk_home]#
[root@localhost sdk_home]#
(6)MindX SDK使用中需要用到OSD功能,安装后需要执行以下命令,生成om文件
bash /usr/local/sdk_home/mxVision/operators/opencvosd/generate_osd_om.sh
执行成功后,显示如下效果
[root@localhost ~]# bash /usr/local/sdk_home/mxVision/operators/opencvosd/generate_osd_om.sh
ASCEND_HOME is set to /usr/local/Ascend by user
Set ASCEND_VERSION to the default value:ascend-toolkit/latest
ATC start working now,please wait for a moment.
ATC run success, welcome to the next use.
The model has been successfully converted to om,please get it under /usr/local/sdk_home/mxVision/operators/opencvosd.
[root@localhost ~]#
(9)安装完MindX SDK后,需要配置环境变量
.bashrc文件添加以下环境变量
# 安装mxVision时配置
. /usr/local/sdk_home/mxVision/set_env.sh
用户也可以通过修改~/.bashrc文件方式设置永久环境变量,操作如下:
a) 以运行用户在任意目录下执行vi ~/.bashrc命令,打开.bashrc文件,在文件最后一行后面添加上述内容。
b) 执行:wq!命令保存文件并退出。
c) 执行source ~/.bashrc命令使其立即生效。
三、ATC模型转换
样例模型快速下载地址:
https://gitee.com/ai_samples/pytorch_models/tree/master/cv/object_detection/yolov5
1、yolov5-2.0获取权重文件。(yolov5-5.0 跳过此步)
(1)pt文件转换为onnx文件
python3.7 pytorch 1.5 建议搭配yolov5 2.0
下载ultralytics-2.0(软件包名为yolov5-2.0.tar.gz)。
wget https://github.com/ultralytics/yolov5/archive/v2.0.tar.gz
运行如下命令,解压软件包并修改转换脚本,目前ATC(om文件转换工具)支持的onnx算子版本为opset11。
tar -xzf yolov5-2.0.tar.gz
vi yolov5-2.0/models/export.py
:set number
i
# 修改第48行opset_version为11
:wq!
在yolov5-2.0目录运行如下命令:
python models/export.py --weights ./yolov5s.pt --img 640 --batch 1
运行结果:生成yolov5s.onnx文件。
(2)onnx文件简化及算子处理
首先对导出的onnx图使用onnx-simplifer工具进行简化。在yolov5-2.0目录运行如下命令:
python -m onnxsim --skip-optimization yolov5s.onnx yolov5s_sim.onnx
运行结果:生成yolov5s_sim.onnx文件。
然后利用附件脚本image_crnn_resnet50_yolov5/data/models/yolov5/modify_yolov5s_2.0_slice.py修改模型Slice算子。将附件脚本上传yolov5-2.0目录,运行如下命令:
python modify_yolov5s_2.0_slice.py yolov5s_sim.onnx
运行结果:生成yolov5s_sim_t.onnx文件。
2、yolov5-5.0获取权重文件。(yolov5-2.0 跳过此步)
(1)pt文件转换为onnx文件
python3.8 pytorch 1.7 建议搭配yolov5 5.0
下载ultralytics-5.0(软件包名为yolov5-5.0.tar.gz)。
wget https://github.com/ultralytics/yolov5/archive/v5.0.tar.gz
运行如下命令,解压软件包并修改转换脚本,目前ATC(om文件转换工具)支持的onnx算子版本为opset11。
tar -xzf yolov5-5.0.tar.gz
vi yolov5-5.0/models/export.py
:set number
i
# 修改第77行opset_version为11
:wq!
在yolov5-5.0目录运行如下命令:
python models/export.py --weights ./yolov5s.pt --img 640 --batch 1
运行结果:生成yolov5s.onnx文件。
(2)onnx文件简化及算子处理
首先对导出的onnx图使用onnx-simplifer工具进行简化。在yolov5-5.0目录运行如下命令:
python -m onnxsim --skip-optimization yolov5s.onnx yolov5s_sim.onnx
运行结果:生成yolov5s_sim.onnx文件。
然后利用附件脚本image_crnn_resnet50_yolov5/data/models/yolov5/modify_yolov5s_5.0_slice.py修改模型Slice算子。将附件脚本上传yolov5-5.0目录,运行如下命令:
python modify_yolov5s_5.0_slice.py yolov5s_sim.onnx
运行结果:生成yolov5s_sim_t.onnx文件。
3、onnx文件转换为om文件
(1)把生成的yolov5s_sim_t.onnx放在image_crnn_resnet50_yolov5/data/models/yolov5目录下
[root@localhost yolov5]#
[root@localhost yolov5]# ls
aipp_yolov5.cfg atc_310.sh atc_310P3.sh coco2014.names modify_yolov5s_2.0_slice.py modify_yolov5s_5.0_slice.py yolov5.cfg yolov5.pt yolov5s_sim_t.onnx
[root@localhost yolov5]#
(2)执行模型转换命令
Ascend310芯片模型转换命令如下:
atc \
--model=./yolov5s_sim_t.onnx \
--framework=5 \
--output=./yolov5s \
--input_format=NCHW \
--input_shape="images:1,3,640,640" \
--enable_small_channel=1 \
--insert_op_conf=./aipp_yolov5.cfg \
--soc_version=Ascend310 \
--log=info
Ascend310P3芯片模型转换命令如下:
atc \
--model=./yolov5s_sim_t.onnx \
--framework=5 \
--output=./yolov5s \
--input_format=NCHW \
--input_shape="images:1,3,640,640" \
--enable_small_channel=1 \
--insert_op_conf=./aipp_yolov5.cfg \
--soc_version=Ascend310P3 \
--log=info
参数说明:
–model:待转换的ONNX模型。
–framework:5代表ONNX模型。
–output:输出的om模型。
–input_format:输入数据的格式。
–input_shape:输入数据的shape。
–insert_op_conf=./aipp_yolov5.cfg:AIPP插入节点,通过config文件配置算子信息,功能包括图片色域转换、裁剪、归一化,主要用于处理原图输入数据,常与DVPP配合使用,详见下文数据预处理。
详细ATC命令转换学习请参考:
https://support.huawei.com/enterprise/zh/doc/EDOC1100234054?idPath=23710424%7C251366513%7C22892968%7C251168373
3、模型转换后,会在目录下生成yolov5s.om
[root@localhost yolov5]#
[root@localhost yolov5]# ls
aipp_yolov5.cfg atc_310.sh atc_310P3.sh coco2014.names modify_yolov5s_2.0_slice.py modify_yolov5s_5.0_slice.py yolov5.cfg yolov5.om yolov5.pt yolov5s_sim_t.onnx
[root@localhost yolov5]#
四、使用image_crnn_resnet50_yolov5
1、修改run_cpp.sh & run_python.sh中MX_SDK_HOME为MindX SDK安装目录
export MX_SDK_HOME=/usr/local/sdk_home/mxVision
2、执行run_cpp.sh 或者 run_python.sh
bash run_cpp.sh
bash run_python.sh
3、文字识别、目标分类、目标检测的结果与预期是一致的
五、image_crnn_resnet50_yolov5详解
1、技术流程图
(1)Pytorch CRNN的文字识别
视频解码:调用OPENCV解码能力,转换为 YUV 格式图像数据。
图像缩放:调用OPENCV,将图像缩放到一定尺寸大小。
模型推理:CRNN模型针对文字进行OCR识别。
模型后处理:针对推理结果进行后处理文字转换。
数据序列化:将stream结果组装成json字符串输出。
(2)Pytorch RESNET50的目标分类
视频解码:调用OPENCV解码能力,转换为 YUV 格式图像数据。
图像缩放:调用OPENCV,将图像缩放到一定尺寸大小。
目标分类:RESTNET50模型针对图像进行目标分类。
模型后处理:针对推理结果进行后处理文字转换。
数据序列化:将stream结果组装成json字符串输出。
(2)Pytorch YoLoV5的目标检测
视频解码:调用DVPP解码能力,转换为 YUV 格式图像数据。
图像缩放:调用DVPP,将图像缩放到一定尺寸大小。
目标检测:YoLoV5模型针对图像进行目标检测。
模型后处理:针对推理结果进行后处理文字转换。
数据序列化:将stream结果组装成json字符串输出。
2、pipeline详解
(1)Pytorch CRNN的文字识别
{
"crnn_ocr": {
"stream_config": { ##设置业务流在哪个芯片上处理
"deviceId": "0"
},
"mxpi_imagedecoder0": { ##图像解码(OpenCV方式)
"props": {
"handleMethod": "opencv"
},
"factory": "mxpi_imagedecoder",
"next": "mxpi_imageresize0"
},
"mxpi_imageresize0": { ##图像缩放(OpenCV方式)
"props": {
"handleMethod": "opencv",
"resizeHeight": "32",
"resizeWidth": "100",
"resizeType": "Resizer_Stretch"
},
"factory": "mxpi_imageresize",
"next": "mxpi_tensorinfer0"
},
"mxpi_tensorinfer0": { ##模型推理
"props": {
"dataSource": "mxpi_imageresize0",
"modelPath": "data/models/crnn/crnn.om", ##模型路径
"waitingTime": "2000",
"outputDeviceId": "-1"
},
"factory": "mxpi_tensorinfer",
"next": "mxpi_classpostprocessor0"
},
"mxpi_classpostprocessor0": { ##模型后处理
"props": {
"dataSource": "mxpi_tensorinfer0",
"postProcessConfigPath": "data/models/crnn/crnn.cfg",
"labelPath": "data/models/crnn/crnn-label.names",
"postProcessLibPath": "libcrnnpostprocess.so"
},
"factory": "mxpi_textgenerationpostprocessor",
"next": "mxpi_dataserialize0"
},
"mxpi_dataserialize0": { ##数据序列化
"props": {
"outputDataKeys": "mxpi_classpostprocessor0"
},
"factory": "mxpi_dataserialize",
"next": "appsink0"
},
"appsrc0": {
"props": {
"blocksize": "409600"
},
"factory": "appsrc",
"next": "mxpi_imagedecoder0"
},
"appsink0": { ##输出推理结果
"props": {
"blocksize": "4096000"
},
"factory": "appsink"
}
}
}
参数说明:
resizeHeight:模型高度,请根据模型的实际尺寸输入。
resizeWidth:模型宽度,请根据模型的实际尺寸输入。
modelPath:模型路径,请根据模型实际路径修改。
postProcessConfigPath:模型配置文件路径,请根据模型配置文件的实际路径修改。
labelPath:标签文件路径,请根据标签文件的实际路径修改。
(2)Pytorch RESNET50的目标分类
{
"resnet50_classification": {
"stream_config": { ##设置业务流在哪个芯片上处理
"deviceId": "0"
},
"appsrc0": {
"props": {
"blocksize": "409600"
},
"factory": "appsrc",
"next": "mxpi_imagedecoder0"
},
"mxpi_imagedecoder0": { ##图像解码(OpenCV方式)
"props": {
"handleMethod": "opencv"
},
"factory": "mxpi_imagedecoder",
"next": "mxpi_imageresize0"
},
"mxpi_imageresize0": { ##图像缩放(OpenCV方式)
"props": {
"handleMethod": "opencv",
"resizeHeight": "304",
"resizeWidth": "304",
"resizeType": "Resizer_Stretch"
},
"factory": "mxpi_imageresize",
"next": "mxpi_opencvcentercrop0"
},
"mxpi_opencvcentercrop0": { ##图像裁剪(OpenCV方式)
"props": {
"dataSource": "mxpi_imageresize0",
"cropHeight": "256",
"cropWidth": "256"
},
"factory": "mxpi_opencvcentercrop",
"next": "mxpi_tensorinfer0"
},
"mxpi_tensorinfer0": { ##模型推理
"props": {
"dataSource": "mxpi_opencvcentercrop0",
"modelPath": "data/models/resnet50/resnet50_pytorch.om", ##模型路径
"waitingTime": "2000",
"outputDeviceId": "-1"
},
"factory": "mxpi_tensorinfer",
"next": "mxpi_classpostprocessor0"
},
"mxpi_classpostprocessor0": { ##模型后处理
"props": {
"dataSource": "mxpi_tensorinfer0",
"postProcessConfigPath": "data/models/resnet50/resnet50_aipp_pt.cfg",
"labelPath": "data/models/resnet50/imagenet1000_clsidx_to_labels.names",
"postProcessLibPath": "libresnet50postprocess.so"
},
"factory": "mxpi_classpostprocessor",
"next": "mxpi_dataserialize0"
},
"mxpi_dataserialize0": { ##数据序列化
"props": {
"outputDataKeys": "mxpi_classpostprocessor0"
},
"factory": "mxpi_dataserialize",
"next": "appsink0"
},
"appsink0": { ##输出推理结果
"props": {
"blocksize": "4096000"
},
"factory": "appsink"
}
}
}
参数说明:
resizeHeight:模型高度,请根据模型的实际尺寸输入。
resizeWidth:模型宽度,请根据模型的实际尺寸输入。
modelPath:模型路径,请根据模型实际路径修改。
postProcessConfigPath:模型配置文件路径,请根据模型配置文件的实际路径修改。
labelPath:标签文件路径,请根据标签文件的实际路径修改。
(3)Pytorch YoLoV5的目标检测
{
"yolov5_detection": {
"stream_config": { ##设置业务流在哪个芯片上处理
"deviceId": "0"
},
"mxpi_imagedecoder0": { ##图像解码(纯硬件)
"factory": "mxpi_imagedecoder",
"next": "mxpi_imageresize0"
},
"mxpi_imageresize0": { ##图像缩放(纯硬件)
"props": {
"parentName": "mxpi_imagedecoder0",
"resizeHeight": "640",
"resizeWidth": "640",
"resizeType": "Resizer_KeepAspectRatio_Fit"
},
"factory": "mxpi_imageresize",
"next": "mxpi_modelinfer0"
},
"mxpi_modelinfer0": { ##模型推理
"props": {
"parentName": "mxpi_imageresize0",
"modelPath": "data/models/yolov5/yolov5s.om", ##模型路径
"postProcessConfigPath": "data/models/yolov5/yolov5.cfg",
"labelPath": "data/models/yolov5/coco2014.names",
"postProcessLibPath": "libMpYOLOv5PostProcessor.so"
},
"factory": "mxpi_modelinfer",
"next": "mxpi_dataserialize0"
},
"mxpi_dataserialize0": { ##数据序列化
"props": {
"outputDataKeys": "mxpi_modelinfer0"
},
"factory": "mxpi_dataserialize",
"next": "appsink0"
},
"appsrc0": {
"props": {
"blocksize": "409600"
},
"factory": "appsrc",
"next": "mxpi_imagedecoder0"
},
"appsink0": { ##输出推理结果
"props": {
"blocksize": "4096000"
},
"factory": "appsink"
}
}
}
参数说明:
modelPath:模型路径,请根据模型实际路径修改。
postProcessConfigPath:模型配置文件路径,请根据模型配置文件的实际路径修改。
labelPath:标签文件路径,请根据标签文件的实际路径修改。
3、C++源码详解
int main(int argc, char* argv[])
{
// 读取pipeline配置文件
std::string CrnnPipCfg = ReadPipelineConfig("data/pipeline/crnn.pipeline");
std::string Resnet50PipCfg = ReadPipelineConfig("data/pipeline/resnet50.pipeline");
std::string Yolov5PipCfg = ReadPipelineConfig("data/pipeline/yolov5.pipeline");
if (CrnnPipCfg == "" || Resnet50PipCfg == "" || Yolov5PipCfg == "") {
LogError << "Read pipeline failed.";
return APP_ERR_COMM_INIT_FAIL;
}
// 初始化 Stream manager 资源
MxStream::MxStreamManager mxStreamManager;
APP_ERROR ret = mxStreamManager.InitManager();
if (ret != APP_ERR_OK) {
LogError << GetError(ret) << "Failed to init Stream manager.";
return ret;
}
// 根据指定的pipeline配置创建多个Stream
APP_ERROR ret1 = mxStreamManager.CreateMultipleStreams(CrnnPipCfg);
APP_ERROR ret2 = mxStreamManager.CreateMultipleStreams(Resnet50PipCfg);
APP_ERROR ret3 = mxStreamManager.CreateMultipleStreams(Yolov5PipCfg);
if (ret1 != APP_ERR_OK || ret2 != APP_ERR_OK || ret3 != APP_ERR_OK) {
LogError << GetError(ret1) << GetError(ret2) << GetError(ret3) << "Failed to create Stream.";
return ret1;
}
//发送测试图片到Stream进行推理
ret1 = InferenceProcess(mxStreamManager, "crnn_ocr", "data/ocr.jpg");
ret2 = InferenceProcess(mxStreamManager, "resnet50_classification", "data/dog.jpg");
ret3 = InferenceProcess(mxStreamManager, "yolov5_detection", "data/bus.jpg");
if (ret1 != APP_ERR_OK || ret2 != APP_ERR_OK || ret3 != APP_ERR_OK) {
LogError << GetError(ret1) << GetError(ret2) << GetError(ret3) << "Failed to Inference Process.";
return ret;
}
// 销毁Stream
mxStreamManager.DestroyAllStreams();
return 0;
}
4、Python源码详解
if __name__ == '__main__':
# 初始化 Stream manager 资源
streamManagerApi = StreamManagerApi()
ret = streamManagerApi.InitManager()
if ret != 0:
print("Failed to init Stream manager, ret=%s" % str(ret))
exit()
# 根据指定的pipeline配置创建Stream
ReadPipelineConfigAndCreateStream("data/pipeline/crnn.pipeline")
ReadPipelineConfigAndCreateStream("data/pipeline/resnet50.pipeline")
ReadPipelineConfigAndCreateStream("data/pipeline/yolov5.pipeline")
# 发送测试图片到Stream进行推理
InferenceProcess(streamManagerApi, b'crnn_ocr', "data/ocr.jpg")
InferenceProcess(streamManagerApi, b'resnet50_classification', "data/dog.jpg")
InferenceProcess(streamManagerApi, b'yolov5_detection', "data/bus.jpg")
# 销毁Stream
streamManagerApi.DestroyAllStreams()