openvino INT8 FP16 FP32 数据类型性能对比

先说结论：

• 在CPU上，INT8略快；在GPU上，FP16最快；
• 使用GPU比使用CPU节约80%左右的CPU开销；
• 使用GPU会比使用CPU占用更多的内存，不知道是不是因为显卡没有现存，共享内存导致的；
• 模型加载时间差异很大，使用GPU可以大大节省CPU资源，但初始化加载模型的过程较慢；

测试环境：intel core i5 8500
应用：human_pose_estimation_demo
模型：human-pose-estimation-0001
数据：MOT16/test/MOT16-06

耗时(ms)	CPU	GPU
FP16-INT8	124559	179275
FP16	128788	81001.2
FP32	127422	103820

模型加载时间：

• CPU FP16-INT8：小于2s；
• GPU FP16：大约53s；

CPU占用率差异：

• CPU FP16-INT8：290%
• GPU FP16：60%

内存占用差异：

• CPU FP16-INT8：VIRT：1.18G，RES：136M
• GPU FP16：VIRT：1.59G，RES：528M

日志：
root@openpose:/openpose_lite# /root/omz_demos_build/intel64/Release/human_pose_estimation_demo -i MOT16/test/MOT16-06/img1/%6d.jpg -no_show -m intel/human-pose-estimation-0001/FP16-INT8/human-pose-estimation-0001.xml
InferenceEngine: 0x7f3ae5583030
Parsing input parameters
Reshape needed
To close the application, press ‘CTRL+C’ here
Total Inference time: 124559

Execution successful
root@openpose:/openpose_lite# /root/omz_demos_build/intel64/Release/human_pose_estimation_demo -i MOT16/test/MOT16-06/img1/%6d.jpg -no_show -m intel/human-pose-estimation-0001/FP16/human-pose-estimation-0001.xml
InferenceEngine: 0x7fbb0eaa2030
Parsing input parameters
Reshape needed
To close the application, press ‘CTRL+C’ here
Total Inference time: 128788

Execution successful
root@openpose:/openpose_lite# /root/omz_demos_build/intel64/Release/human_pose_estimation_demo -i MOT16/test/MOT16-06/img1/%6d.jpg -no_show -m intel/human-pose-estimation-0001/FP32/human-pose-estimation-0001.xml
InferenceEngine: 0x7f9136f9f030
Parsing input parameters
Reshape needed
To close the application, press ‘CTRL+C’ here
Total Inference time: 127422

Execution successful
root@openpose:/openpose_lite# /root/omz_demos_build/intel64/Release/human_pose_estimation_demo -i MOT16/test/MOT16-06/img1/%6d.jpg -no_show -d GPU -m intel/human-pose-estimation-0001/FP32/human-pose-estimation-0001.xml
InferenceEngine: 0x7f1a2209e030
Parsing input parameters
Reshape needed
To close the application, press ‘CTRL+C’ here
Total Inference time: 103820

Execution successful
root@openpose:/openpose_lite# /root/omz_demos_build/intel64/Release/human_pose_estimation_demo -i MOT16/test/MOT16-06/img1/%6d.jpg -no_show -d GPU -m intel/human-pose-estimation-0001/FP16/human-pose-estimation-0001.xml
InferenceEngine: 0x7fe9f13a0030
Parsing input parameters
Reshape needed
To close the application, press ‘CTRL+C’ here
Total Inference time: 81001.2

Execution successful
root@openpose:/openpose_lite# /root/omz_demos_build/intel64/Release/human_pose_estimation_demo -i MOT16/test/MOT16-06/img1/%6d.jpg -no_show -d GPU -m intel/human-pose-estimation-0001/FP16-INT8/human-pose-estimation-0001.xml
InferenceEngine: 0x7fe0ce749030
Parsing input parameters
Reshape needed
To close the application, press ‘CTRL+C’ here
Total Inference time: 179275

Execution successful