Triton Server 快速入门

已于 2023-04-10 10:34:14 修改
阅读量3k 收藏 13
点赞数 3
分类专栏： 机器学习 python 深度学习 文章标签： 机器学习 深度学习 人工智能 triton
于 2023-04-04 11:53:13 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/xzpdxz/article/details/129858857
版权

在这里插入图片描述
官方文档

背景

  • 在工业场景中，常常阻碍模型部署的不是模型本身，而是算力原因，
  • 许多高精度的模型，都有一个比较大的参数量
  • Triton server 是英伟达Nvidia开源的高性能推理，可以在CPU、GPU上加速模型推理的一个工具

是什么

  • triton是一个模型推理服务工具
  • 具有动态批处理，并发执行，模型集成和串流输入，使用配置方式实现模型pipline
  • 使用脚本方式充当模型，以便使计算过程用在显存中
  • triton server 服务对外可以提供api-GRPC/HTTP，以及导出 Prometheus 指标，用于监控 GPU 利用率、延迟、内存使用率和推理吞吐量
  • 可以使用triton client 发送推理请求
  • Triton 支持一些主流加速推理框架ONNXRuntime、TensorFlow SavedModel 和 TensorRT 后端
  • Triton支持深度学习，机器学习，逻辑回归等学习模型
  • Triton 支持基于GPU，x86,ARM CPU，除此之外支持国产GCU（需要安装GCU的ONNXRUNTIME）
  • 模型可在生成环境中实时更新，无需重启Triton Server
  • Triton 支持对单个 GPU 显存无法容纳的超大模型进行多 GPU 以及多节点推理
  • 支持性能评估，包括GPU利用率、server吞吐量和server延迟时间

代码实例

准备onnx 模型

  • 我们需要提前训练好自己onnx模型文件，这里我们使用官方提供好的onnx模型文件
# Create model repository with placeholder for model and version 1
mkdir -p ./models/densenet_onnx/1

# Download model and place it in model repository
wget -O ./models/densenet_onnx/1/model.onnx https://contentmamluswest001.blob.core.windows.net/content/14b2744cf8d6418c87ffddc3f3127242/9502630827244d60a1214f250e3bbca7/08aed7327d694b8dbaee2c97b8d0fcba/densenet121-1.2.onnx

创建一个最小化的模型配置

vim ./models/densenet_onnx/config.pbtxt

name: "densenet_onnx"
backend: "onnxruntime"
max_batch_size: 0
input: [
  {
    name: "data_0",
    data_type: TYPE_FP32,
    dims: [ 1, 3, 224, 224]
  }
]
output: [
  {
    name: "fc6_1",
    data_type: TYPE_FP32,
    dims: [ 1, 1000, 1, 1 ]
  }
]

这里定义模型的输入输出可以通过netron工具可视化查看该模型的输入输出参数

拉取两个官方镜像

docker pull nvcr.io/nvidia/tritonserver:23.02-py3  # triton server
docker pull nvcr.io/nvidia/tritonserver:23.02-py3-sdk # triton client

启动triton server

  • 启动容器
# Start server container in the background
docker run -d --gpus=all --network=host -v $PWD:/mnt --name triton-server nvcr.io/nvidia/tritonserver:23.02-py3 bash
  • 进入容器并运行
[~]# tritonserver --model-repository=/mnt/models --model-control-mode=poll
I0403 06:07:10.866992 1186 server.cc:522] 
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I0403 06:07:10.867083 1186 server.cc:549] 
+-------------+-------------------------------------------------------------------------+--------+
| Backend     | Path                                                                    | Config |
+-------------+-------------------------------------------------------------------------+--------+
| pytorch     | /opt/tritonserver/backends/pytorch/libtriton_pytorch.so                 | {}     |
| tensorflow  | /opt/tritonserver/backends/tensorflow1/libtriton_tensorflow1.so         | {}     |
| onnxruntime | /opt/tritonserver/backends/onnxruntime/libtriton_onnxruntime.so         | {}     |
| openvino    | /opt/tritonserver/backends/openvino_2021_2/libtriton_openvino_2021_2.so | {}     |
+-------------+-------------------------------------------------------------------------+--------+

I0403 06:07:10.867131 1186 server.cc:592] 
+---------------+---------+--------+
| Model         | Version | Status |
+---------------+---------+--------+
| densenet_onnx | 2       | READY  |
+---------------+---------+--------+

I0403 06:07:10.947730 1186 metrics.cc:623] Collecting metrics for GPU 0: NVIDIA GeForce RTX 3090
I0403 06:07:10.947760 1186 metrics.cc:623] Collecting metrics for GPU 1: NVIDIA GeForce RTX 3090
I0403 06:07:10.947772 1186 metrics.cc:623] Collecting metrics for GPU 2: NVIDIA GeForce RTX 3090
I0403 06:07:10.947784 1186 metrics.cc:623] Collecting metrics for GPU 3: NVIDIA GeForce RTX 3090
I0403 06:07:10.947800 1186 metrics.cc:623] Collecting metrics for GPU 4: NVIDIA GeForce RTX 3090
I0403 06:07:10.947819 1186 metrics.cc:623] Collecting metrics for GPU 5: NVIDIA GeForce RTX 3090
I0403 06:07:10.947852 1186 metrics.cc:623] Collecting metrics for GPU 6: NVIDIA GeForce RTX 3090
I0403 06:07:10.947886 1186 metrics.cc:623] Collecting metrics for GPU 7: NVIDIA GeForce RTX 3090
I0403 06:07:10.949215 1186 tritonserver.cc:1932] 
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option                           | Value                                                                                                                                                                                        |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id                        | triton                                                                                                                                                                                       |
| server_version                   | 2.19.0                                                                                                                                                                                       |
| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |
| model_repository_path[0]         | /mnt/models                                                                                                                                                                                  |
| model_control_mode               | MODE_POLL                                                                                                                                                                                    |
| strict_model_config              | 1                                                                                                                                                                                            |
| rate_limit                       | OFF                                                                                                                                                                                          |
| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |
| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |
| cuda_memory_pool_byte_size{1}    | 67108864                                                                                                                                                                                     |
| cuda_memory_pool_byte_size{2}    | 67108864                                                                                                                                                                                     |
| cuda_memory_pool_byte_size{3}    | 67108864                                                                                                                                                                                     |
| cuda_memory_pool_byte_size{4}    | 67108864                                                                                                                                                                                     |
| cuda_memory_pool_byte_size{5}    | 67108864                                                                                                                                                                                     |
| cuda_memory_pool_byte_size{6}    | 67108864                                                                                                                                                                                     |
| cuda_memory_pool_byte_size{7}    | 67108864                                                                                                                                                                                     |
| response_cache_byte_size         | 0                                                                                                                                                                                            |
| min_supported_compute_capability | 6.0                                                                                                                                                                                          |
| strict_readiness                 | 1                                                                                                                                                                                            |
| exit_timeout                     | 30                                                                                                                                                                                           |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

I0403 06:07:10.950873 1186 grpc_server.cc:4375] Started GRPCInferenceService at 0.0.0.0:8001
I0403 06:07:10.951176 1186 http_server.cc:3075] Started HTTPService at 0.0.0.0:8000
I0403 06:07:10.992539 1186 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

1.日志中显示densenet_onnx 模型已加载完毕，并启动了GRPC、HTTP、Metrics接口
2.–model-control-mode=poll该参数用于启动模型热更新，当模型文件发生变化，或者新增版本时，程序先启动新的实例版本出来，在将旧版本或者实例卸载掉

  • 新增模型版本
[~]# cp -rf 1 2
[~]# cp -rf 1 3

I0403 06:07:26.109494 1186 onnxruntime.cc:2400] TRITONBACKEND_ModelInitialize: densenet_onnx (version 3)
I0403 06:07:26.119616 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 0)
I0403 06:07:26.319224 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 1)
I0403 06:07:26.495285 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 2)
I0403 06:07:26.669370 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 3)
I0403 06:07:26.829762 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 4)
I0403 06:07:27.007662 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 5)
I0403 06:07:27.182506 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 6)
I0403 06:07:27.367420 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 7)
I0403 06:07:27.532531 1186 model_repository_manager.cc:1149] successfully loaded 'densenet_onnx' version 3
I0403 06:07:27.532561 1186 model_repository_manager.cc:1026] unloading: densenet_onnx:2
I0403 06:07:27.532729 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:07:27.548199 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:07:27.561028 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:07:27.573967 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:07:27.585593 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:07:27.596050 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:07:27.605498 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:07:27.614892 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:07:27.624120 1186 onnxruntime.cc:2423] TRITONBACKEND_ModelFinalize: delete model state
I0403 06:07:27.624158 1186 model_repository_manager.cc:1132] successfully unloaded 'densenet_onnx' version 2
I0403 06:14:42.551308 1186 model_repository_manager.cc:994] loading: densenet_onnx:3
I0403 06:14:42.651625 1186 onnxruntime.cc:2400] TRITONBACKEND_ModelInitialize: densenet_onnx (version 3)
I0403 06:14:42.659502 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 0)
I0403 06:14:42.851975 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 1)
I0403 06:14:43.027086 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 2)
I0403 06:14:43.203822 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 3)
I0403 06:14:43.378325 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 4)
I0403 06:14:43.552427 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 5)
I0403 06:14:43.732855 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 6)
I0403 06:14:43.903087 1186 onnxruntime.cc:2443] TRITONBACKEND_ModelInstanceInitialize: densenet_onnx (GPU device 7)
I0403 06:14:44.071766 1186 model_repository_manager.cc:1149] successfully loaded 'densenet_onnx' version 3
I0403 06:14:44.071795 1186 model_repository_manager.cc:1026] unloading: densenet_onnx:3
I0403 06:14:44.071970 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:14:44.081007 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:14:44.089658 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:14:44.098768 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:14:44.107905 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:14:44.116819 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:14:44.125697 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:14:44.134503 1186 onnxruntime.cc:2477] TRITONBACKEND_ModelInstanceFinalize: delete instance state
I0403 06:14:44.143469 1186 onnxruntime.cc:2423] TRITONBACKEND_ModelFinalize: delete model state
I0403 06:14:44.143503 1186 model_repository_manager.cc:1132] successfully unloaded 'densenet_onnx' version 3

上述结果中将新增版本2，3，最终将triton server版本换到3

  • 整体运行实例截图
    在这里插入图片描述

客户端压测

  • 启动客户端
docker run -itd --gpus=all --network=host -v $PWD:/mnt --name triton-client nvcr.io/nvidia/tritonserver:23.02-py3-sdk bash
  • 使用perf_analyzer压测
[~]# perf_analyzer -m densenet_onnx -u 127.0.0.1:8000 --concurrency-range 1:6

Inferences/Second vs. Client Average Batch Latency
Concurrency: 1, throughput: 96.1522 infer/sec, latency 10396 usec
Concurrency: 2, throughput: 197.181 infer/sec, latency 10138 usec
Concurrency: 3, throughput: 305.046 infer/sec, latency 9832 usec
Concurrency: 4, throughput: 425.759 infer/sec, latency 9392 usec
Concurrency: 5, throughput: 564.87 infer/sec, latency 8850 usec
Concurrency: 6, throughput: 704.574 infer/sec, latency 8514 usec

模型分析工具

  • 进入triton-server 容器中
tritonserver --model-repository=/mnt/models --model-control-mode=explicit # 必须使用这个模式否则无法load/unload模型
  • 进入triton-client容器中
root@53:/mnt# cat config.yaml # 使用remote方式的原因，可以通过这种方式进行测试当我们不需要GPU时，来测试国产芯片的吞吐以及延迟
model_repository: /mnt/models
#checkpoint_directory: /mnt/checkpoints/
profile_models: densenet_onnx
triton_grpc_endpoint: 127.0.0.1:9001
triton_metrics_url: 127.0.0.1:9002
triton_launch_mode: remote

root@53:/mnt# rm -rf output_model_repository/ checkpoints/ && model-analyzer profile -f config.yaml  # 这里选择使用配置文件方式
[Model Analyzer] Initializing GPUDevice handles
[Model Analyzer] Using GPU 0 NVIDIA GeForce RTX 3090 with UUID GPU-b6d3bb44-b607-e9c1-c898-3977340c20a4
[Model Analyzer] Using GPU 1 NVIDIA GeForce RTX 3090 with UUID GPU-f37fdb1b-77c7-ff1f-21c0-e2db53fe0818
[Model Analyzer] Using GPU 2 NVIDIA GeForce RTX 3090 with UUID GPU-1a0e40f7-65eb-9694-f91c-253808416e71
[Model Analyzer] Using GPU 3 NVIDIA GeForce RTX 3090 with UUID GPU-c889529d-734f-8a13-f820-02597663a704
[Model Analyzer] Using GPU 4 NVIDIA GeForce RTX 3090 with UUID GPU-9f08b528-c421-bc60-2fc6-7f906e13404a
[Model Analyzer] Using GPU 5 NVIDIA GeForce RTX 3090 with UUID GPU-9c9fbba1-0558-4f8e-1534-8ff8e8b03a6c
[Model Analyzer] Using GPU 6 NVIDIA GeForce RTX 3090 with UUID GPU-55808174-5a3e-8082-8759-b248794a1e34
[Model Analyzer] Using GPU 7 NVIDIA GeForce RTX 3090 with UUID GPU-2a0fd91b-3ca8-8249-2d0c-70c7853491a6
[Model Analyzer] Using remote Triton Server
[Model Analyzer] WARNING: GPU memory metrics reported in the remote mode are not accurate. Model Analyzer uses Triton explicit model control to load/unload models. Some frameworks do not release the GPU memory even when the memory is not being used. Consider using the "local" or "docker" mode if you want to accurately monitor the GPU memory usage for different models.
[Model Analyzer] WARNING: Config sweep parameters are ignored in the "remote" mode because Model Analyzer does not have access to the model repository of the remote Triton Server.
[Model Analyzer] No checkpoint file found, starting a fresh run.
[Model Analyzer] Profiling server only metrics...
[Model Analyzer] Profiling densenet_onnx: client batch size=1, concurrency=1
[Model Analyzer] Profiling densenet_onnx: client batch size=1, concurrency=2
[Model Analyzer] Profiling densenet_onnx: client batch size=1, concurrency=4
[Model Analyzer] Profiling densenet_onnx: client batch size=1, concurrency=8
[Model Analyzer] Profiling densenet_onnx: client batch size=1, concurrency=16
[Model Analyzer] Profiling densenet_onnx: client batch size=1, concurrency=32
[Model Analyzer] Profiling densenet_onnx: client batch size=1, concurrency=64
[Model Analyzer] Profiling densenet_onnx: client batch size=1, concurrency=128
[Model Analyzer] No longer increasing concurrency as throughput has plateaued
[Model Analyzer] Saved checkpoint to /mnt/checkpoints/0.ckpt
[Model Analyzer] Profile complete. Profiled 1 configurations for models: ['densenet_onnx']
[Model Analyzer] 
[Model Analyzer] WARNING: GPU output field "gpu_used_memory", has no data
[Model Analyzer] WARNING: GPU output field "gpu_utilization", has no data
[Model Analyzer] WARNING: GPU output field "gpu_power_usage", has no data
[Model Analyzer] WARNING: Server output field "gpu_used_memory", has no data
[Model Analyzer] WARNING: Server output field "gpu_utilization", has no data
[Model Analyzer] WARNING: Server output field "gpu_power_usage", has no data
[Model Analyzer] Exporting inference metrics to /mnt/results/metrics-model-inference.csv
[Model Analyzer] WARNING: Requested top 3 configs, but found only 1. Showing all available configs for this model.
[Model Analyzer] WARNING: Requested top 3 configs, but found only 1. Showing all available configs for this model.
[Model Analyzer] WARNING: Requested top 3 configs, but found only 1. Showing all available configs for this model.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] Exporting Summary Report to /mnt/reports/summaries/densenet_onnx/result_summary.pdf
[Model Analyzer] WARNING: Requested top 3 configs, but found only 1. Showing all available configs for this model.
[Model Analyzer] WARNING: Requested top 3 configs, but found only 1. Showing all available configs for this model.
[Model Analyzer] To generate detailed reports for the 1 best configurations, run `model-analyzer report --report-model-configs densenet_onnx --export-path /mnt --config-file config.yaml

root@53:/mnt# ls
checkpoints  config.yaml  models  models23  output_model_repository  plots  reports  results
root@53:/mnt# cat results/metrics-
metrics-model-inference.csv  metrics-server-only.csv      
root@53:/mnt# cat results/metrics-model-inference.csv 
Model,Batch,Concurrency,Model Config Path,Instance Group,Max Batch Size,Satisfies Constraints,Throughput (infer/sec),p99 Latency (ms)
densenet_onnx,1,16,densenet_onnx,8:GPU,0,Yes,1394.9,13.1
densenet_onnx,1,64,densenet_onnx,8:GPU,0,Yes,1384.2,50.0
densenet_onnx,1,32,densenet_onnx,8:GPU,0,Yes,1384.0,25.3
densenet_onnx,1,128,densenet_onnx,8:GPU,0,Yes,1331.5,104.6
densenet_onnx,1,8,densenet_onnx,8:GPU,0,Yes,1215.6,7.7
densenet_onnx,1,4,densenet_onnx,8:GPU,0,Yes,472.0,12.2
densenet_onnx,1,2,densenet_onnx,8:GPU,0,Yes,172.5,17.3
densenet_onnx,1,1,densenet_onnx,8:GPU,0,Yes,95.6,15.6
  • 生成可视化文档
root@53:/mnt# model-analyzer report --report-model-configs densenet_onnx --export-path /mnt --config-file config.yaml
[Model Analyzer] Loaded checkpoint from file /mnt/checkpoints/0.ckpt
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_power_usage' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_power_usage' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_power_usage' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_power_usage' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_power_usage' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_power_usage' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_power_usage' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_power_usage' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_used_memory' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] WARNING: No GPU metric corresponding to tag 'gpu_utilization' found in the model's measurement. Possibly comparing measurements across devices.
[Model Analyzer] Exporting Detailed Report to /mnt/reports/detailed/densenet_onnx/detailed_report.pdf

在这里插入图片描述

  • model-analyzer profile 参数
root@sse-lg-113-53:/mnt# model-analyzer profile --help
usage: model-analyzer profile [-h] [-f CONFIG_FILE] [-s CHECKPOINT_DIRECTORY] [-i MONITORING_INTERVAL] [-d DURATION_SECONDS] [--collect-cpu-metrics] [--gpus GPUS] [--skip-summary-reports]
                              [-m MODEL_REPOSITORY] [--output-model-repository-path OUTPUT_MODEL_REPOSITORY_PATH] [--override-output-model-repository] [-r CLIENT_MAX_RETRIES]
                              [--client-protocol {http,grpc}] [--profile-models PROFILE_MODELS] [-b BATCH_SIZES] [-c CONCURRENCY] [--reload-model-disable] [--perf-analyzer-timeout PERF_ANALYZER_TIMEOUT]
                              [--perf-analyzer-cpu-util PERF_ANALYZER_CPU_UTIL] [--perf-analyzer-path PERF_ANALYZER_PATH] [--perf-output] [--perf-output-path PERF_OUTPUT_PATH]
                              [--perf-analyzer-max-auto-adjusts PERF_ANALYZER_MAX_AUTO_ADJUSTS] [--triton-launch-mode {local,docker,remote,c_api}] [--triton-docker-image TRITON_DOCKER_IMAGE]
                              [--triton-http-endpoint TRITON_HTTP_ENDPOINT] [--triton-grpc-endpoint TRITON_GRPC_ENDPOINT] [--triton-metrics-url TRITON_METRICS_URL] [--triton-server-path TRITON_SERVER_PATH]
                              [--triton-output-path TRITON_OUTPUT_PATH] [--triton-docker-mounts TRITON_DOCKER_MOUNTS] [--triton-docker-shm-size TRITON_DOCKER_SHM_SIZE]
                              [--triton-install-path TRITON_INSTALL_PATH] [--early-exit-enable] [--run-config-search-max-concurrency RUN_CONFIG_SEARCH_MAX_CONCURRENCY]
                              [--run-config-search-min-concurrency RUN_CONFIG_SEARCH_MIN_CONCURRENCY] [--run-config-search-max-instance-count RUN_CONFIG_SEARCH_MAX_INSTANCE_COUNT]
                              [--run-config-search-min-instance-count RUN_CONFIG_SEARCH_MIN_INSTANCE_COUNT] [--run-config-search-max-model-batch-size RUN_CONFIG_SEARCH_MAX_MODEL_BATCH_SIZE]
                              [--run-config-search-min-model-batch-size RUN_CONFIG_SEARCH_MIN_MODEL_BATCH_SIZE] [--run-config-search-mode {brute,quick}] [--run-config-search-disable]
                              [--run-config-profile-models-concurrently-enable] [-e EXPORT_PATH] [--filename-model-inference FILENAME_MODEL_INFERENCE] [--filename-model-gpu FILENAME_MODEL_GPU]
                              [--filename-server-only FILENAME_SERVER_ONLY] [--num-configs-per-model NUM_CONFIGS_PER_MODEL] [--num-top-model-configs NUM_TOP_MODEL_CONFIGS]
                              [--inference-output-fields INFERENCE_OUTPUT_FIELDS] [--gpu-output-fields GPU_OUTPUT_FIELDS] [--server-output-fields SERVER_OUTPUT_FIELDS] [--latency-budget LATENCY_BUDGET]
                              [--min-throughput MIN_THROUGHPUT]

optional arguments:
  -h, --help            show this help message and exit
  -f CONFIG_FILE, --config-file CONFIG_FILE
                        Path to Config File for subcommand 'profile'.
  -s CHECKPOINT_DIRECTORY, --checkpoint-directory CHECKPOINT_DIRECTORY
                        Full path to directory to which to read and write checkpoints and profile data.
  -i MONITORING_INTERVAL, --monitoring-interval MONITORING_INTERVAL
                        Interval of time between metrics measurements in seconds
  -d DURATION_SECONDS, --duration-seconds DURATION_SECONDS
                        Specifies how long (seconds) to gather server-only metrics
  --collect-cpu-metrics
                        Specify whether CPU metrics are collected or not
  --gpus GPUS           List of GPU UUIDs to be used for the profiling. Use 'all' to profile all the GPUs visible by CUDA.
  --skip-summary-reports
                        Skips the generation of analysis summary reports and tables.
  -m MODEL_REPOSITORY, --model-repository MODEL_REPOSITORY
                        Triton Model repository location
  --output-model-repository-path OUTPUT_MODEL_REPOSITORY_PATH
                        Output model repository path used by Model Analyzer. This is the directory that will contain all the generated model configurations
  --override-output-model-repository
                        Will override the contents of the output model repository and replace it with the new results.
  -r CLIENT_MAX_RETRIES, --client-max-retries CLIENT_MAX_RETRIES
                        Specifies the max number of retries for any requests to Triton server.
  --client-protocol {http,grpc}
                        The protocol used to communicate with the Triton Inference Server
  --profile-models PROFILE_MODELS
                        List of the models to be profiled
  -b BATCH_SIZES, --batch-sizes BATCH_SIZES
                        Comma-delimited list of batch sizes to use for the profiling
  -c CONCURRENCY, --concurrency CONCURRENCY
                        Comma-delimited list of concurrency values or ranges <start:end:step> to be used during profiling
  --reload-model-disable
                        Flag to indicate whether or not to disable model loading and unloading in remote mode.
  --perf-analyzer-timeout PERF_ANALYZER_TIMEOUT
                        Perf analyzer timeout value in seconds.
  --perf-analyzer-cpu-util PERF_ANALYZER_CPU_UTIL
                        Maximum CPU utilization value allowed for the perf_analyzer.
  --perf-analyzer-path PERF_ANALYZER_PATH
                        The full path to the perf_analyzer binary executable
  --perf-output         Enables the output from the perf_analyzer to a file specified by perf_output_path. If perf_output_path is None, output will be written to stdout.
  --perf-output-path PERF_OUTPUT_PATH
                        Path to the file to which write perf_analyzer output, if enabled.
  --perf-analyzer-max-auto-adjusts PERF_ANALYZER_MAX_AUTO_ADJUSTS
                        Maximum number of times perf_analyzer is launched with auto adjusted parameters in an attempt to profile a model.
  --triton-launch-mode {local,docker,remote,c_api}
                        The method by which to launch Triton Server. 'local' assumes tritonserver binary is available locally. 'docker' pulls and launches a triton docker container with the specified
                        version. 'remote' connects to a running server using given http, grpc and metrics endpoints. 'c_api' allows direct benchmarking of Triton locallywithout the use of endpoints.
  --triton-docker-image TRITON_DOCKER_IMAGE
                        Triton Server Docker image tag
  --triton-http-endpoint TRITON_HTTP_ENDPOINT
                        Triton Server HTTP endpoint url used by Model Analyzer client.
  --triton-grpc-endpoint TRITON_GRPC_ENDPOINT
                        Triton Server HTTP endpoint url used by Model Analyzer client.
  --triton-metrics-url TRITON_METRICS_URL
                        Triton Server Metrics endpoint url.
  --triton-server-path TRITON_SERVER_PATH
                        The full path to the tritonserver binary executable
  --triton-output-path TRITON_OUTPUT_PATH
                        The full path to the file to which Triton server instance will append their log output. If not specified, they are not written.
  --triton-docker-mounts TRITON_DOCKER_MOUNTS
                        A list of strings representing volumes to be mounted. The strings should have the format '<host path>:<container path>:<access mode>'.
  --triton-docker-shm-size TRITON_DOCKER_SHM_SIZE
                        The size of the /dev/shm for the triton docker container
  --triton-install-path TRITON_INSTALL_PATH
                        Path to Triton install directory i.e. the parent directory of 'lib/libtritonserver.so'.Required only when using triton_launch_mode=c_api.
  --early-exit-enable   Flag to indicate if Model Analyzer can skip some configurations when manually searching concurrency or max_batch_size
  --run-config-search-max-concurrency RUN_CONFIG_SEARCH_MAX_CONCURRENCY
                        Max concurrency value that run config search should not go beyond that.
  --run-config-search-min-concurrency RUN_CONFIG_SEARCH_MIN_CONCURRENCY
                        Min concurrency value that run config search should start with.
  --run-config-search-max-instance-count RUN_CONFIG_SEARCH_MAX_INSTANCE_COUNT
                        Max instance count value that run config search should not go beyond that.
  --run-config-search-min-instance-count RUN_CONFIG_SEARCH_MIN_INSTANCE_COUNT
                        Min instance count value that run config search should start with.
  --run-config-search-max-model-batch-size RUN_CONFIG_SEARCH_MAX_MODEL_BATCH_SIZE
                        Value for the model's max_batch_size that run config search will not go beyond.
  --run-config-search-min-model-batch-size RUN_CONFIG_SEARCH_MIN_MODEL_BATCH_SIZE
                        Value for the model's max_batch_size that run config search will start from.
  --run-config-search-mode {brute,quick}
                        The search mode for Model Analyzer to find and evaluate model configurations. 'brute' will brute force all combinations of configuration options. 'quick' will attempt to find a
                        near-optimal configuration as fast as possible, but isn't guaranteed to find the best.
  --run-config-search-disable
                        Disable run config search.
  --run-config-profile-models-concurrently-enable
                        Enable the profiling of all supplied models concurrently.
  -e EXPORT_PATH, --export-path EXPORT_PATH
                        Full path to directory in which to store the results
  --filename-model-inference FILENAME_MODEL_INFERENCE
                        Specifies filename for storing model inference metrics
  --filename-model-gpu FILENAME_MODEL_GPU
                        Specifies filename for storing model GPU metrics
  --filename-server-only FILENAME_SERVER_ONLY
                        Specifies filename for server-only metrics
  --num-configs-per-model NUM_CONFIGS_PER_MODEL
                        The number of configurations to plot per model in the summary.
  --num-top-model-configs NUM_TOP_MODEL_CONFIGS
                        Model Analyzer will compare this many of the top models configs across all models.
  --inference-output-fields INFERENCE_OUTPUT_FIELDS
                        Specifies column keys for model inference metrics table
  --gpu-output-fields GPU_OUTPUT_FIELDS
                        Specifies column keys for model gpu metrics table
  --server-output-fields SERVER_OUTPUT_FIELDS
                        Specifies column keys for server-only metrics table
  --latency-budget LATENCY_BUDGET
                        Shorthand flag for specifying a maximum latency in ms.
  --min-throughput MIN_THROUGHPUT
                        Shorthand flag for specifying a minimum throughput.

总体架构图

在这里插入图片描述

整体处理流程

  • Model Repository表示模型仓库，可以是本地的，也可以是在线的，用于存储Triton将会读取的模型（也就是示例代码中的挂载目录）；
  • Triton通过HTTP/gRPC协议接收外部请求，或者是直接通过调用C API接收外部请求，然后把外部请求的推理数据发送给每一个模型scheduler，模型scheduler对请求数据进行最优化组批，并根据其管理的各个模型的类型将数据发送到对应的backend；
  • 每个深度学习框架的backend接收请求数据，执行推理过程，获取结果；
  • Triton再把所有的推理结果返回给client端；

开发

  • 可扩充：Triton支持C API的后端，可以自定义预处理和后处理操作，甚至可以兼容新的深度学习框架；
  • 模型管理API：模型管理API可以查询和控制Triton管理的模型；
  • 状态监控：可以实时查看各终端是否就绪、是否在运行及其利用率，总体的吞吐量和延迟；

并发模型执行

  • Triton结构允许多个模型或单/多个模型的多个实例并发执行
    在这里插入图片描述
  • 如果Triton管理的一个模型同时接收到了多个请求，那么Triton会对这些请求进行序列化排序，一次只执行一个
    在这里插入图片描述

Triton提供了一个叫做instance-group的模型配置项，允许指定每一个模型允许的并发实例的数量,这些并发的模型数量称之为一个instance。默认情况下，Triton是一个GPU上放一个模型，一次只推理一份数据。但通过设置模型的instance_group参数，可以对模型的并发实例数据量进行扩充
在这里插入图片描述

三种模型调度策略

stateless

stateful

ensemble 集成模型

  • ensemble模型表示把多个模型组成一个pipeline进行处理，把内部包含的前面模型的输出作为下一个模型的输入，例如定义图像预处理 -> 推理 -> 图像后处理过程
  • ensemble模型必须使用ensemble scheduler，不需要考虑ensemble model管理的各个子model的scheduler，一个ensemble model并不是一个真正的模型，只是用来指明各个子model之间的数据流动路径
name: "ensemble_model"
platform: "ensemble"
max_batch_size: 1
input [
  {
    name: "IMAGE"
    data_type: TYPE_STRING
    dims: [ 1 ]
  }
]
output [
  {
    name: "CLASSIFICATION"
    data_type: TYPE_FP32
    dims: [ 1000 ]
  },
  {
    name: "SEGMENTATION"
    data_type: TYPE_FP32
    dims: [ 3, 224, 224 ]
  }
]
ensemble_scheduling {
  step [
    {
      model_name: "image_preprocess_model"
      model_version: -1
      input_map {
        key: "RAW_IMAGE"
        value: "IMAGE"
      }
      output_map {
        key: "PREPROCESSED_OUTPUT"
        value: "preprocessed_image"
      }
    },
    {
      model_name: "classification_model"
      model_version: -1
      input_map {
        key: "FORMATTED_IMAGE"
        value: "preprocessed_image"
      }
      output_map {
        key: "CLASSIFICATION_OUTPUT"
        value: "CLASSIFICATION"
      }
    },
    {
      model_name: "segmentation_model"
      model_version: -1
      input_map {
        key: "FORMATTED_IMAGE"
        value: "preprocessed_image"
      }
      output_map {
        key: "SEGMENTATION_OUTPUT"
        value: "SEGMENTATION"
      }
    }
  ]
}
  • 流程图
    在这里插入图片描述

模型仓库

  • 启动 triton server 服务时，我们需要指定加载模型的路径
  • tritonserver --model-repository=xxx

本地路径

  • tritonserver --model-repository=/path/to/model/repository

S3对象存储

  • tritonserver --model-repository=s3://bucket/path/to/model/repository

其他

  • Google Cloud Storage、Amazon S3和Azure Storage

模型类型

  • TensorRT模型：model.plan
  • Onnx模型：model.onnx
  • TorchScript模型：model.pt
  • Tensorflow模型：model.graphdef 或 model.savedmodel
  • OpenVINO模型：model.xml model.bin
  • Python模型：model.py
  • Dali模型：model.dali

模型管理模式

NONE模式

  • Triton启动时会尝试加载模型仓库中的所有模型，对于无法加载的模型，会标识为UNAVAILABLE且无法用于推理；
  • Triton运行过程中，对模型仓库的修改会被忽略。调用模型控制API发送的请求无效且会收到错误的Response；
  • 在启动Triton是设置–model-control-mode=none会启用本模式，None模式也是默认的模型管理模式。

EXPLICIT模式

  • Triton启动时仅会加载标注了–load-model命令行的模型，如果所有的模型都没有指定–load-model，那么就不会加载任何一个模型。对于无法加载的模型，会标识为UNAVAILABLE且无法用于推理；
  • Triton运行过程中，调用模型控制API可以进行模型的加载和卸载，返回的response的状态用于标识加载和卸载行为是否成功。当尝试重新加载一个已经加载了的模型时，如果重新加载失败了则不会对已加载的模型造成影响。如果重新加载成功了，则会使用新加载的模型。
  • 在启动Triton是设置–model-control-mode=explicit会启用本模式；

POLL模式

  • 在启动Triton是设置–model-control-mode=poll会启用本模式，通过设置–repository-poll-secs为非零值设置poll的时间间隔
  • 该模式是一个热加载模式，文件变动或者新增版本都可以触发重新加载新版本模型

模型配置说明

platform

  • triton的backend，表示使用什么类型推理

max_batch_size

input、output

  • 定义输入输出
max_batch_size: 0
input: [
  {
    name: "data_0",
    data_type: TYPE_FP32,
    dims: [ 1, 3, 224, 224]
  }
]
output: [
  {
    name: "prob_1",
    data_type: TYPE_FP32,
    dims: [ 1, 1000, 1, 1 ]
  }
]

name

  • 默认为model所在的文件夹的名字

version_policy

  • 指定可用模型的版本信息
  • 默认为version_policy: { latest: { num_versions: 1}}，即使用最新的一个模型
  • model repo中的所有版本的模型都可用；version_policy: { all: {}}

instance_group

  • 设定model的instance数量，以支持并发响应外部数据的推理请求；
instance_group [
	{
	count: 2
	kind: KIND_GPU #每个GPU上创建2个instance
	
	}
]

instance_group [
	{
	count: 1
	kind: KIND_GPU
	gpus: [ 0 ]
	},
	{
	count: 2
	kind: KIND_GPU
	gpus: [ 1, 2 ]
	}
] #gpu0上创建一个instance，gpu1和gpu2上各创建两个instance

instance_group [
	{
	count: 2
	kind: KIND_CPU
	}
] #在CPU上创建两个instance

rate_limiter

instance_group [
    {
      count: 1
      kind: KIND_GPU
      gpus: [ 0, 1, 2 ]
      rate_limiter {
        resources [
          {
            name: "R1"
            count: 4
          },
          {
            name: "R2"
            global: True
            count: 2
          }
        ]
        priority: 2
      }
    }
  ]
【trition-server】pytorch 文档：使用 Triton 提供 Torch-TensorRT 模型
突围
07-05 221
Serving a Torch-TensorRT model with Triton
AI模型工业部署：综述【常用的部署框架：TensorRT、Libtorch】【常见提速方法：模型结构、剪枝、蒸馏、量化训练、稀疏化】【常见部署流程：onnx2trt】【常见服务部署搭配】
u013250861的博客
05-08 6467
作为深度学习算法工程师，训练模型和部署模型是最基本的要求，每天都在重复着这个工作，但偶尔静下心来想一想，还是有很多事情需要做的： 模型的结构，因为上线业务需要，更趋向于稳定有经验的，而不是探索一些新的结构 模型的加速仍然不够，还没有压榨完GPU的全部潜力 一、AI部署 AI部署的基本步骤： 训练一个模型，也可以是拿一个别人训练好的模型 针对不同平台对生成的模型进行转换，也就是俗称的parse、convert，即前端解释器 针对转化后的模型进行优化，这一步很重要，涉及到很多优化的步骤 在特定的平台(嵌入
服务器：Triton Inference Server提供了优化的云和边缘推理解决方案
02-04
Triton推理服务器 最新版本：您目前在master分支上，该分支跟踪开发进度到下一发行版。 Triton Inference Server的最新版本是2.5.0，可在分支上。 Triton Inference Server提供了针对CPU和GPU优化的云和边缘推理解决方案。 Triton支持HTTP / REST和GRPC协议，该协议允许远程客户端为服务器管理的任何模型请求推理。 对于边缘部署，Triton可以作为具有C API的共享库使用，该API允许Triton的全部功能直接包含在应用程序中。 Triton Inference Server的当前版本为2.5.0，与上的triton
指南：Triton Inference Server
01-24 3412
关于triton inference server的详细记录，涉及安装，服务部署，优化。通过例子一步一步搭建一个可实际使用的服务。
深度学习部署：Triton（Triton inference server）【旧称：TensorRT serving，专门针对TensorRT设计的服务器框架，后来变为Triton，支持其他推理后端】
u013250861的博客
08-01 2360
一般来说，我们都是从最主要的server开始编，编译的时候会链接core、common、backend中的代码，其他自定义backend（比如tensorrt_backend）在编译的时候也需要带上common、core、backend这三个仓库，这些关系我们可以从相应的CMakeList中找到。官方也提供了客户端，C++和python的都可以有，可以直接使用官方的，也可以根据官方提供的http协议构造自己的客户端，只要会构造body，一切都很简单。就可以发送请求，结果也会传回response里。
tritonserver学习之一：triton使用流程
liupenglove的专栏
12-27 2400
进入client镜像，测试tritonserver推理。
tritonserver学习之九：tritonserver grpc异步模式
最新发布
liupenglove的专栏
03-11 2311
gRPC的异步模式允许服务器同时处理多个客户端请求，而不会因等待某个请求的响应而阻塞其他请求的处理。这使得TritonServer能够充分利用系统资源，提高并发性能，从而能够更高效地处理大量的模型推理请求。：在异步模式下，服务器不会为每个请求创建单独的线程或进程，而是将请求放入队列中，并通过事件循环机制来处理这些请求。这减少了系统资源的开销，使得TritonServer能够在有限的资源下处理更多的请求。
固件安全测试入门学习手册 （新手必看）
systemino的博客
04-25 4388
无论是公网连接还是独立网络，固件都是控制嵌入式设备的核心。因此，必须要了解如何分析固件以执行未授权的功能。要开始进行安全测试和固件逆向，在即将进行评估时，请使用以下方法作为指导。该方法包括九个阶段，旨在使安全研究人员，软件开发人员和信息安全专业人员能够进行固件安全测试研究。 以下各节将在适用的情况下通过支持示例进一步详细介绍每个阶段。 原文报告如下： https://scriptin...
OpenCV调用YOLOv5模型ONNX：从入门到精通
[OpenCV调用YOLOv5模型ONNX：从入门到精通](https://opengraph.githubassets.com/7e5c7f98ed1706722dc036b8fd1d5ea0ddb572ac24c70430cdbbc90452e70081/xun-xh/yolov5-onnx-pyqt-exe) # 1. OpenCV与YOLOv5模型简介 ...
Triton inference server教程1
lansebingxuan的专栏
06-29 2177
本系列文章是Triton Inference Server部署的详细教程，教程1介绍triton的整体架构以及Model Repository目录文件的准备
TRITON - Web Security
07-08
websence的Triton软件用于提供webproxy
基于Triton Server部署BERT模型
ljp1919的专栏
03-11 6964
背景 本文简要介绍如何使用 Triton 部署 BERT模型，主要参考 NVIDIA/DeepLearningExamples 准备工作 下载数据 进入到/data/DeepLearningExamples-master/PyTorch/LanguageModeling/BERT/data/squad后，下载数据: bash ./squad_download.sh 下载模型 wget --content-disposition https://api.ngc.nvidia.com/v2/models/n
triton-tritonserver的main函数
evsqiezi
09-15 649
tritonserver是triton的远端服务器。
深度学习模型服务端部署：Triton server+TensorRT【PyTorch格式----＞ONNX格式----＞TensorRT格式----＞利用Triton server部署】
u013250861的博客
08-01 1818
本来是想着直接用简单的Flask+TensorRT去做服务端的，但是奈何我比较懒，现在已经沉溺于Python，再去花费时间学一个Flask我估摸着时间不够，于是百度之后发现了Tritonserver，这个是NVIDIA自己做的开源产品，支持GRPC和HTTP，还说支持C API直接访问（这部分不了解，相关视频后续在官方教程里会给出）sid=493256最后说一下为什么要选择TensorRT。
tritonserver学习之二：tritonserver编译
liupenglove的专栏
01-09 1464
【代码】tritonserver学习之三：tritonserver源码编译。
Nvidia DeepStream和Triton Server学习经验分享
u011268946的博客
06-30 1219
介绍DeepStream和Triton Server
ubuntu 20.04 安装triton server
zhoutianyou的专栏
10-25 1157
triton server
NVIDIA之Triton Inference Server环境部署安装
热门推荐
ZONGXP的博客
09-27 1万+
0 背景 Triton是什么？Triton是 NVIDIA 推出的 Inference Server，专门做 AI 模型的部署服务。客户端可以同伙HTTP/REST或gRPC的方式来请求服务，特性包括以下方面： 支持多种框架，例如 Tensorflow、TensoRT、Pytorch、ONNX甚至自定义框架后端； 支持 GPU 和 CPU 方式运行，能最大化利用硬件资源； 容器化部署，集成 k8s，可以方便的进行编排和扩展； 支持并发模型，支持多种模型或同一模型的不同实例在同一GPU上运行 支持
nvdia triton server运行pt文件
柴犬犬舍
08-20 1263
triton server默认都是tensorrt推理。 但也会出现有操作不支持，导致无法转模型为engine的情况。 可以选择直接运行pytorch的pt文件，以下为操作步骤。 1.pytorch后端环境编译步骤 原理是使用 pytorch C++ API运行pt文件模型。 安装依赖项 apt-get install patchelf rapidjson-dev python3-dev 构建NGC 的 PyTorch 容器。 例如，构建一个使用 NGC 21.02 版本的 PyTor.
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值