移动端推理引擎通用性小测

移动端推理引擎通用性小测

推理引擎

引擎	公司	Commit id(或Tag)	链接	使用方法
MNN	阿里巴巴	2.6.0	https://github.com/tensorflow/tensorflow/tree/v2.13.0-rc1	https://mnn-docs.readthedocs.io/en/latest/start/overall.html
TensorFlow Lite	谷歌	v2.13.0-rc1	https://github.com/tensorflow/tensorflow/tree/v2.13.0-rc1	https://tensorflow.google.cn/lite/performance/benchmarks?hl=zh-cn
Paddle Lite	百度	v2.13-rc	https://github.com/PaddlePaddle/Paddle-Lite/tree/v2.13-rc	https://github.com/PaddlePaddle/Paddle-Lite/blob/v2.13-rc/README.md
NCNN	腾讯	20230517	https://github.com/Tencent/ncnn/tree/20230517	https://github.com/Tencent/ncnn/wiki/how-to-build
TNN		fbe15fe6e7eb3858fd356b6d189a48821d2e9d49	https://github.com/Tencent/TNN/tree/fbe15fe6e7eb3858fd356b6d189a48821d2e9d49	https://github.com/Tencent/TNN/blob/fbe15fe6e7eb3858fd356b6d189a48821d2e9d49/doc/cn/get_started.md
MindSpore Lite	华为	v2.0.0-rc1	https://github.com/mindspore-ai/mindspore/tree/v2.0.0-rc1	https://www.mindspore.cn/lite/docs/en/master/use/build.html

测试模型

类别		模型	地址	输入尺寸
视觉	分类	MobileNet-V2	https://github.com/onnx/models/blob/main/vision/classification/mobilenet/model/mobilenetv2-10.onnx	1,3,224,224
		ResNet50-V1	https://github.com/onnx/models/blob/main/vision/classification/resnet/model/resnet50-v1-7.onnx	1,3,224,224
		SqueezeNet-V1.1	https://github.com/onnx/models/blob/main/vision/classification/squeezenet/model/squeezenet1.1-7.onnx	1,3,224,224
	检测分割	FasterRCNN-12	https://github.com/onnx/models/blob/main/vision/object_detection_segmentation/faster-rcnn/model/FasterRCNN-12.onnx	3,224,224
	超分辨率	SuperResolution-10	https://github.com/onnx/models/blob/main/vision/super_resolution/sub_pixel_cnn_2016/model/super-resolution-10.onnx	1,1,224,224
文本	GPT	GPT-2	https://github.com/onnx/models/blob/main/text/machine_comprehension/gpt-2/model/gpt2-10.onnx	1,8,56
	Bert	Bertsquad-12	https://github.com/onnx/models/blob/main/text/machine_comprehension/bert-squad/model/bertsquad-12.onnx	input_ids:0=1,256 input_mask:0=1,256 segment_ids:0=1,256 unique_ids_raw_output___9:0=1

模型支持情况

CPU 运行

模型	MNN	TensorFlow Lite	Paddle Lite	TNN	MindSpore Lite	NCNN
MobileNet-V2	O	O	O	X	O	O
ResNet50-V1	O	O	O	O	O	O
SqueezeNet-V1.1	O	O	O	O	O	O
FasterRCNN-12	O	X	X	X	X	X
SuperResolution-10	O	O	O	O	O	O
GPT-2	O	X	X	X	O	X
Bertsquad-12	O	X	X	X	X	X

GPU 运行 (OpenCL 或 Vulkan)

模型	MNN	TensorFlow Lite	Paddle Lite	TNN	MindSpore Lite	NCNN
MobileNet-V2	O	X	O	X	O	O
ResNet50-V1	O	X	O	O	O	O
SqueezeNet-V1.1	O	O	O	O	O	O
FasterRCNN-12	X	X	X	X	X	X
SuperResolution-10	O	O	O	X	O	O
GPT-2	O	X	X	X	X	X
Bertsquad-12	O	X	X	X	X	X

结论与后续

	CPU	GPU	视觉	文本
MNN	7/7	6/7	5/5	2/2
Paddle-Lite	4/7	4/7	4/5	0/2
TNN	3/7	3/7	3/5	0/2
Mindspore-Lite	5/7	3/7	4/5	1/2
NCNN	4/7	4/7	4/5	0/2
Tensorflow-Lite	4/7	2/7	4/5	0/2

• MNN 全部模型均支持CPU运行，但GPU 跑 FasterRCNN12 出现错误（已于 2.6.2 修正）
• 其他推理引擎对 视觉类模型支持较好，文本类模型支持较弱。
• 带控制流的ONNX模型仅 MNN 转换运行成功