How to predict fast? 修改onnx model支持batch模式

 在onnxruntime中如何可以支持“batching”模式使模型前向运行更快呢：

For example, if your model has graph input (taking image case) of 
shape [1, 3, 224, 224] 
- it can only take one 3 channel image of height and width 224. 
But if it has graph inputlike this - ["some_string", 3, 224, 224]
- it can handle "batched" input of any size.

上述意思是修改onnx模型input的第一个维度为None即可适应batch输入，此外，需要对应修改output的第一个维度为None

import onnx
mp = onnx.load_model('aa.onnx')
mp.graph.input[0].type.tensor_type.shape.dim[0].dim_param = 'None'
onnx.save(mp, 'aa_batch.onnx')

[C++] after session_ load aa_batch.onnx.
auto output_tensors = session_->Run(Ort::RunOptions{ nullptr }, input_node_names.data(), &input_tensor, 1, output_node_names.data(), 1);
It runs very fast! 

eg：

import onnx
from onnx import helper, checker
from onnx import TensorProto,GraphProto
import onnx.utils
from onnx.tools import update_model_dims
import re
import argparse
import numpy as np

onnx_path='./resnet50-v1-7.onnx'
model = onnx.load(onnx_path)                         
                                                      
dim_proto0 = model.graph.input[0].type.tensor_type.shape.dim[0]
dim_proto0.dim_param = 'None'
dim_proto_o1 = model.graph.output[0].type.tensor_type.shape.dim[0]
dim_proto_o1.dim_param = 'None'
onnx.checker.check_model(model)
onnx.save(model, 'dynamic_input_resnet_batch_50.onnx')

参考链接：https://github.com/microsoft/onnxruntime/issues/2118
                  OnnxRunTime的部署流程_hjxu2016的博客-CSDN博客_c++ onnx 部署