nvidia-docker run -it --rm nvcr.io/nvidia/tensorrtserver:x.x-py3
1.2 deploy
nvidia-docker run --rm -p8000:8000 -p8001:8001 -v/path/to/examples/models:/models nvcr.io/nvidia/tensorrtserver:x.x-py3 trtserver --model-store=/models
1.3 inspect
host ip:port/api/status, Warming you can inspect the states of models, such as ready_state, and if ready_state==MODEL_UNAVAILABLE, trtis will not recognize this model.
Example
1.1 build a yolov3 model
Warming
if you get any error when running yolov3_to_onnx.py, try to reduce onnx version to 1.4.1. Like this:
pip uninstall onnx
pip install onnx==1.4.1
there are two ways to build yolov3 tensorrt engine, one is build it in the docker, another is in the /usr/local/TensorRT-5.1.5.0/sample/python/yolov3_to_onnx, the latter will be easier.
1.1.1 build in a docker image
follow the guide to build a yolov3 tensorrt engine
start a tensorrt container
docker run \
-v $PWD/trt:/workspace/trt \
--name trt \
-ti nvcr.io/nvidia/tensorrt:19.10-py2 /bin/bash
build yolov3 model
# inside container trtexport TRT_PATH=/usr/src/tensorrt
cd$TRT_PATH/samples/python/yolov3_onnx/;
pip installwget
pip install onnx==1.5.0
# will automatic download the model and convert into onnx
python yolov3_to_onnx.py;# build trtexec engine cd$TRT_PATH/samples/trtexec;make;cd../../;
./bin/trtexec --onnx=$TRT_PATH/samples/python/yolov3_onnx/yolov3.onnx --saveEngine=$TRT_PATH/model.plan
# Average over 10 runs is 30.8623 ms (host walltime is 31.4395 ms, 99% percentile time is 31.9949)
copy model
# at your hostmkdir -p $model_path/yolov3_608_trt/1
docker cp trt:/usr/src/tensorrt/model.plan $model_path/yolov3_608_trt/1
1.1.2 build in TensorRT 5.1.5.0 (python3)
move to TensorRT-path/samples/python/yolov3_onnx/
pip install wget
pip install onnx==1.4.1
python yolov3_to_onnx.py
# build trtexec engine cd$TRT_PATH/samples/trtexec;make;cd../../;
./bin/trtexec --onnx=$TRT_PATH/samples/python/yolov3_onnx/yolov3.onnx --saveEngine=$TRT_PATH/model.plan
# Average over 10 runs is 30.8623 ms (host walltime is 31.4395 ms, 99% percentile time is 31.9949)