triton客户端使用

model_analyzer

简介：

Triton Model Analyzer is a CLI tool which can help you find a more optimal configuration, on a given piece of hardware, for single, multiple, ensemble, or BLS models running on a Triton Inference Server. Model Analyzer will also generate reports to help you better understand the trade-offs of the different configurations along with their compute and memory requirements.

代码地址：https://github.com/triton-inference-server/model_analyzer

测试方法：

启docker

docker run --name fan-triton -it --gpus device=0 \

      -v /var/run/docker.sock:/var/run/docker.sock \

      -v $(pwd)/examples/quick-start:$(pwd)/examples/quick-start \

      -v /home/fanz/thirdparty/triton:/home/fanz/thirdparty/triton \

      --net=host nvcr.io/nvidia/tritonserver:23.04-py3-sdk

测试

    /usr/local/bin/model-analyzer profile \

    --model-repository /home/xx/thirdparty/triton/models \

    --profile-models Primary_Detector --triton-launch-mode=docker \

    --output-model-repository-path /home/xx/thirdparty/triton/prm-out2 \

--export-path .

models下包含Primary_Detector等模型文件。Primary_Detector下包含模型配置文件config.pbtxt和模型版本目录1,1下就是模型的engine文件。

跑的时间会很长，因为在运行所有的配置。最后生成如下几个文件，具体意义详见文档：https://github.com/triton-inference-server/model_analyzer/blob/main/docs/report.md

 

perf_analyzer

perf_analyzer是个功能强大的测试工具，支持http, grpc, capi三种模式的测试。http, grpc方式需要启动tritonserver, 客户端发送命令包给服务端，服务端调triton的底层接口进行推理。capi方式不需要启动tritonserver，程序直接调接口进行推理。

perf_analyzer的源码：client/src/c++/perf_analyzer at main · triton-inference-server/client · GitHub。

capi方式的使用：

./perf_analyzer --service-kind=triton_c_api --triton-server-directory=/opt/tritonserver  --model-repository=/opt/nvidia/deepstream/deepstream/samples/models  -m Primary_Detector

model-repository的意义详见上一章节。