openvino 将onnx转为IR并进行int8量化

环境

- Ubuntu 22.04
- python 3.10

安装环境

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python3-venv build-essential python3-dev git-all -y
sudo apt-get install intel-opencl-icd -y

编译 mo

下载 openvino

git clone https://github.com/openvinotoolkit/openvino

编译 mo

cd openvino/tools
python3 -m pip install mo

编译成功输出如下信息：

Collecting mo
  Downloading mo-0.3.0-py2.py3-none-any.whl (12 kB)
Requirement already satisfied: PyYAML in /usr/local/lib/python3.10/dist-packages (from mo) (6.0)
Collecting colorama (from mo)
  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Requirement already satisfied: toml in /usr/local/lib/python3.10/dist-packages (from mo) (0.10.2)
Installing collected packages: colorama, mo
Successfully installed colorama-0.4.6 mo-0.3.0

安装

pip install openvino-dev

查看帮助

mo -h

onnx 转为 IR

mo --input_model onnx/model.onnx --compress_to_fp16 --output_dir ir_model

这里压缩为fp16

输出信息：

[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False.
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11.
Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /workspace/bert/ir_model/model.xml
[ SUCCESS ] BIN file: /workspace/bert/ir_model/model.bin

转换成功了！

输出文件对比：

# ls -lh ir_model/
total 321M
-rw-r--r-- 1 root root 319M Sep 22 11:27 model.bin
-rw-r--r-- 1 root root 1.8M Sep 22 11:27 model.xml

# ls -lh onnx/
total 640M
-rw-r--r-- 1 root root 640M Sep 21 20:23 model.onnx

IR 模型量化为 int8

编译 Post-Training Optimization Tool

cd openvino/tools/pot/
python3 setup.py install

bert模型量化步骤参考：
https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/105-language-quantize-bert

Quantization of Image Classification model参考：
https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/301-tensorflow-training-openvino

参考

1. https://github.com/openvinotoolkit/openvino_notebooks#-installation-guide
2. https://docs.openvino.ai/2022.3/notebooks/102-pytorch-onnx-to-openvino-with-output.html
3. https://github.com/openvinotoolkit/openvino/tree/master/tools/mo
4. https://github.com/openvinotoolkit/openvino
5. https://github.com/openvinotoolkit/openvino/tree/master/tools/pot