基于Triton Server部署BERT模型

已于 2022-11-03 14:33:37 修改
阅读量6.7k 收藏 13
点赞数
分类专栏： 深度学习 模型部署 NLP 文章标签： bert 深度学习 推理加速 模型部署
于 2022-03-11 16:17:32 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/ljp1919/article/details/123426640
版权
深度学习 同时被 3 个专栏收录
55 篇文章 3 订阅
订阅专栏
NLP
37 篇文章 10 订阅
订阅专栏
模型部署
3 篇文章 1 订阅
订阅专栏

更多、更及时内容欢迎留意微信公众号： 小窗幽记机器学习

文章目录

PS: 更多、更及时内容欢迎留意微信公众号： 小窗幽记机器学习

背景

本文简要介绍如何使用 Triton 部署 BERT模型，主要参考 NVIDIA/DeepLearningExamples

准备工作

下载数据

进入到/data/DeepLearningExamples-master/PyTorch/LanguageModeling/BERT/data/squad后，下载数据:

bash ./squad_download.sh

下载模型

wget --content-disposition https://api.ngc.nvidia.com/v2/models/nvidia/bert_large_pyt_amp_ckpt_squad_qa1_1/versions/1/zip -O bert_large_pyt_amp_ckpt_squad_qa1_1_1.zip

在这里插入图片描述

由于各个脚本使用的是bert_qa.pt，所以，对上述模型文件进行重命名。

构建容器

bash ./scripts/docker/build.sh

在这里插入图片描述

Processing triggers for libc-bin (2.27-3ubuntu1) ...
Removing intermediate container 89010b0a75b2
 ---> 562bcc14dbfa
Step 15/15 : COPY . .
 ---> 23bac3585a43
Successfully built 23bac3585a43
Successfully tagged bert:latest

模型部署

将 checkpoint 导出为 torchscript

在宿主机(不需要容器内部)下，进入DeepLearningExamples-master/PyTorch/LanguageModeling/BERT执行下述脚本将 checkpoint 转为 torchscript:

bash ./triton/export_model.sh

转换过程状态：

=============
== PyTorch ==
=============

NVIDIA Release 20.06 (build 13419386)
PyTorch Version 1.6.0a0+9907a3e

Container image Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.

Copyright (c) 2014-2020 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.

NOTE: Legacy NVIDIA Driver detected.  Compatibility mode ENABLED.

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

deploying model bertQA-ts-script in format pytorch_libtorch
/opt/conda/lib/python3.6/site-packages/torch/jit/_recursive.py:160: UserWarning: 'bias' was found in ScriptModule constants,  but it is a non-constant parameter. Consider removing it.
  " but it is a non-constant {}. Consider removing it.".format(name, hint))

conversion correctness test results
-----------------------------------
maximal absolute error over dataset (L_inf):  0.0322265625

average L_inf error over output tensors:  0.02264404296875
variance of L_inf error over output tensors:  5.4970383644104004e-05
stddev of L_inf error over output tensors:  0.00741420148391612

time of error check of native model:  0.8040032386779785 seconds
time of error check of ts model:  1.7353665828704834 seconds

done

模型格式转换后，待部署的Triton模型将存于BERT/results/triton_models。

在这里插入图片描述

./triton/export_model.sh中EXPORT_FORMAT值为ts-script表示转为torchscript格式。如果想要以ONNX格式部署，则可以将./triton/export_model.sh中的EXPORT_FORMAT值设置为onnx。此外，还要注意相应改动triton_model_name，比如改为bertQA-onnx，以对新转换的模型进行合适命名。

在这里插入图片描述

启动 Triton server

可以通过执行以下命令来启动Triton server:

docker run --rm --gpus device=0 --ipc=host --network=host -p 8000:8000 -p 8001:8001 -p 8002:8002 -v $PWD/results/triton_models:/models nvcr.io/nvidia/tritonserver:20.06-v1-py3 trtserver --model-store=/models --log-verbose=1

由于上述镜像nvcr.io/nvidia/tritonserver:20.06-v1-py3本地尚未拉取，所以执行上述命令后，会优先拉取该镜像。

另外，注意这里指定的模型位置是--model-store=/models映射的是./results/triton_models，且该目录下有2个模型，所以服务启动的时候会将2个模型都加载：

在这里插入图片描述

服务启动后，可以看下显存的占用情况：

在这里插入图片描述

启动自定义的Triton client

./triton/client.py为自定义的client代码。

Step1: 启动一个 client 容器

docker run -it --rm --ipc=host --network=host -v $PWD/vocab:/workspace/bert/vocab bert:latest

在这里插入图片描述

PS:
启动客户端无需指定GPU，且上述的启动方式，当在终端直接退出该容器后，该容器自动销毁。

如此便启动了一个容器，并进入容器当中。

Step2: 启动 client
进入到 client 代码目录:cd /workspace/bert/triton/，再运行如下代码，对 bertQA-ts-script 版模型进行请求：

python client.py --do_lower_case --version_2_with_negative --vocab_file=../vocab/vocab --triton-model-name=bertQA-ts-script

在这里插入图片描述

此时，client 端将向已在运行的 Triton server 发送一个请求，Triton server 接收请求并处理后，将请求返回。如果想输入自定义的文本段落和问题，则只需在运行client.py脚本时搭配--question和--context参数并传入对应的内容。此外，可以通过--triton-model-name指定特定的模型。这里服务端加载了2个模型，所以client也可以对 onnx 版模型进行请求：

python client.py --do_lower_case --version_2_with_negative --vocab_file=../vocab/vocab --triton-model-name=bertQA-onnx

在这里插入图片描述

模型部署后的评估：Squad1.1

部署并评估模型，可以在宿主机下执行以下命令

bash ./triton/evaluate.sh

PS:
在部署和评测之前，先将之前启动的 Triton server 关闭，否则端口被冲突。

服务启动和评测运行状态如下:

=============
== PyTorch ==
=============

NVIDIA Release 20.06 (build 13419386)
PyTorch Version 1.6.0a0+9907a3e

Container image Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.

Copyright (c) 2014-2020 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.

NOTE: Legacy NVIDIA Driver detected.  Compatibility mode ENABLED.

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

deploying model bert_large_fp32 in format pytorch_libtorch
/opt/conda/lib/python3.6/site-packages/torch/jit/_recursive.py:160: UserWarning: 'bias' was found in ScriptModule constants,  but it is a non-constant parameter. Consider removing it.
  " but it is a non-constant {}. Consider removing it.".format(name, hint))

conversion correctness test results
-----------------------------------
maximal absolute error over dataset (L_inf):  1.4185905456542969e-05

average L_inf error over output tensors:  1.0482966899871826e-05
variance of L_inf error over output tensors:  8.773056355456296e-12
stddev of L_inf error over output tensors:  2.961934562993635e-06

time of error check of native model:  1.596167802810669 seconds
time of error check of ts model:  2.414717435836792 seconds

done
Starting server...
Waiting for TRITON Server to be ready at http://localhost:8000...
000
.......TRITON Server is ready!

=============
== PyTorch ==
=============

NVIDIA Release 20.06 (build 13419386)
PyTorch Version 1.6.0a0+9907a3e

Container image Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.

Copyright (c) 2014-2020 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.

NOTE: Legacy NVIDIA Driver detected.  Compatibility mode ENABLED.

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

Sending Requests: 100%|███████████████████████████████████████████████████████████████████████████| 10833/10833 [04:20<00:00, 27.84sentences/s-----------------------------█████████████████████████████████████████████████████████████████████▉| 10832/10833 [14:29<00:00, 12.28sentences/s]
Individual Time Runs
Total Time: 869886.3623142242 ms
-----------------------------
-----------------------------
Total Inference Time = 432310.23 forSentences processed = 10833
Throughput Average (sentences/sec) = 12.45
Throughput Average (batches/sec) = 1.56
-----------------------------
-----------------------------
Summary Statistics
Batch size = 8
Sequence Length = 384
Latency Confidence Level 95 (ms) = 594040.61627388
Latency Confidence Level 99 (ms)  = 615392.275094986
Latency Confidence Level 100 (ms)  = 619993.6480522156
Latency Average (ms)  = 319048.1366518239
-----------------------------
Sending Requests: 100%|███████████████████████████████████████████████████████████████████████████| 10833/10833 [15:16<00:00, 11.82sentences/s]
Processed Requests: 100%|█████████████████████████████████████████████████████████████████████████| 10833/10833 [15:16<00:00, 11.82sentences/s]

=============
== PyTorch ==
=============

NVIDIA Release 20.06 (build 13419386)
PyTorch Version 1.6.0a0+9907a3e

Container image Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.

Copyright (c) 2014-2020 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.

NOTE: Legacy NVIDIA Driver detected.  Compatibility mode ENABLED.

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

trt_server_cont
tritonnet

需要注意的是，默认下以torchscript格式部署服务，并以Squad1.1数据集进行评测。如果想对onnx格式模型进行评测，将/triton/evaluate.sh中的EXPORT_FORMAT值从ts-script改为onnx。

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.

NOTE: Legacy NVIDIA Driver detected.  Compatibility mode ENABLED.

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

deploying model bert_large_fp32 in format onnxruntime_onnx
/opt/conda/lib/python3.6/site-packages/torch/onnx/utils.py:955: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input input__0
  'Automatically generated names will be applied to each dynamic axes of input {}'.format(key))
/opt/conda/lib/python3.6/site-packages/torch/onnx/utils.py:955: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input input__1
  'Automatically generated names will be applied to each dynamic axes of input {}'.format(key))
/opt/conda/lib/python3.6/site-packages/torch/onnx/utils.py:955: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input input__2
  'Automatically generated names will be applied to each dynamic axes of input {}'.format(key))
/opt/conda/lib/python3.6/site-packages/torch/onnx/utils.py:955: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input output__0
  'Automatically generated names will be applied to each dynamic axes of input {}'.format(key))
/opt/conda/lib/python3.6/site-packages/torch/onnx/utils.py:955: UserWarning: No names were found for specified dynamic axes of provided input.Automatically generated names will be applied to each dynamic axes of input output__1
  'Automatically generated names will be applied to each dynamic axes of input {}'.format(key))
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 1336539136

conversion correctness test results
-----------------------------------
maximal absolute error over dataset (L_inf):  0.00022530555725097656

average L_inf error over output tensors:  0.0001377016305923462
variance of L_inf error over output tensors:  6.448256743378049e-09
stddev of L_inf error over output tensors:  8.030103824595327e-05

time of error check of native model:  1.2507586479187012 seconds
time of error check of onnx model:  76.80649089813232 seconds

done
Starting server...
Waiting for TRITON Server to be ready at http://localhost:8000...
000
.......TRITON Server is ready!

=============
== PyTorch ==
=============

NVIDIA Release 20.06 (build 13419386)
PyTorch Version 1.6.0a0+9907a3e

Container image Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.

Copyright (c) 2014-2020 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.

NOTE: Legacy NVIDIA Driver detected.  Compatibility mode ENABLED.

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

Sending Requests: 100%|███████████████████████████████████████████████████████████████████████████| 10833/10833 [04:40<00:00, 15.52sentences/s-----------------------------█████████████████████████████████████████████████████████████████████▉| 10832/10833 [14:23<00:00, 12.42sentences/s]
Individual Time Runs
Total Time: 863938.3265972137 ms
-----------------------------
-----------------------------
Total Inference Time = 418017.89 forSentences processed = 10833
Throughput Average (sentences/sec) = 12.54
Throughput Average (batches/sec) = 1.57
-----------------------------
-----------------------------
Summary Statistics
Batch size = 8
Sequence Length = 384
Latency Confidence Level 95 (ms) = 568533.2419872284
Latency Confidence Level 99 (ms)  = 591532.5634479523
Latency Confidence Level 100 (ms)  = 595446.0487365723
Latency Average (ms)  = 308500.2912194087
-----------------------------
Sending Requests: 100%|███████████████████████████████████████████████████████████████████████████| 10833/10833 [15:10<00:00, 11.90sentences/s]
Processed Requests: 100%|█████████████████████████████████████████████████████████████████████████| 10833/10833 [15:10<00:00, 11.90sentences/s]

=============
== PyTorch ==
=============

NVIDIA Release 20.06 (build 13419386)
PyTorch Version 1.6.0a0+9907a3e

Container image Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.

Copyright (c) 2014-2020 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying project or file.

NOTE: Legacy NVIDIA Driver detected.  Compatibility mode ENABLED.

NOTE: MOFED driver for multi-node communication was not detected.
      Multi-node communication performance may be reduced.

trt_server_cont
tritonnet

更多、更及时内容欢迎留意微信公众号： 小窗幽记机器学习

JasonLiu1919
关注
  • 0
    点赞
  • 踩
  • 13
    收藏
    觉得还不错? 一键收藏
  • 3
    评论
专栏目录
深度学习模型服务端部署：Triton server+TensorRT【PyTorch格式----＞ONNX格式----＞TensorRT格式----＞利用Triton server部署】
u013250861的博客
08-01 1615
本来是想着直接用简单的Flask+TensorRT去做服务端的，但是奈何我比较懒，现在已经沉溺于Python，再去花费时间学一个Flask我估摸着时间不够，于是百度之后发现了Tritonserver，这个是NVIDIA自己做的开源产品，支持GRPC和HTTP，还说支持C API直接访问（这部分不了解，相关视频后续在官方教程里会给出）sid=493256最后说一下为什么要选择TensorRT。
yolov4-triton-tensorrt:该存储库将YOLOv4作为优化的TensorRT引擎部署到Triton Inference Server
03-21
Triton Inference Server具有许多现成的优势，可用于模型部署，例如GRPC和HTTP接口，在多个GPU上自动调度，共享内存（甚至在GPU上），运行状况度量和内存资源管理。 TensorRT将通过融合层并为我们的特定硬件选择最...
Triton Server 快速入门
xzpdxz的博客
04-04 2690
Triton提供了一个叫做instance-group的模型配置项，允许指定每一个模型允许的并发实例的数量,这些并发的模型数量称之为一个instance。默认情况下，Triton是一个GPU上放一个模型，一次只推理一份数据。但通过设置模型的instance_group参数，可以对模型的并发实例数据量进行扩充。2.–model-control-mode=poll该参数用于启动模型热更新，当模型文件发生变化，或者新增版本时，程序先启动新的实例版本出来，在将旧版本或者实例卸载掉。
AI模型部署：Triton+TensorRT部署Bert文本向量化服务实践
2401_85375298的博客
05-31 1021
推理服务器和推理后端介绍TensorRT+Triton环境搭建Bert模型转化为ONNX中间表示ONNX中间表示编译为TensorRT模型文件Triton服务端参数配置Triton服务端代码实现Triton服务端启动HTTP客户端请求TensorRT前后压测结果对比在Triton+TensorRT的组合中，Triton是推理服务器，TensorRT是推理后端。
tritonserver学习之九：tritonserver grpc异步模式
liupenglove的专栏
03-11 1612
gRPC的异步模式允许服务器同时处理多个客户端请求，而不会因等待某个请求的响应而阻塞其他请求的处理。这使得TritonServer能够充分利用系统资源，提高并发性能，从而能够更高效地处理大量的模型推理请求。：在异步模式下，服务器不会为每个请求创建单独的线程或进程，而是将请求放入队列中，并通过事件循环机制来处理这些请求。这减少了系统资源的开销，使得TritonServer能够在有限的资源下处理更多的请求。
Triton inference server教程1
lansebingxuan的专栏
06-29 2071
本系列文章是Triton Inference Server部署的详细教程，教程1介绍triton的整体架构以及Model Repository目录文件的准备
指南：Triton Inference Server
01-24 2196
关于triton inference server的详细记录，涉及安装，服务部署，优化。通过例子一步一步搭建一个可实际使用的服务。
tritonserver学习之二：tritonserver编译
liupenglove的专栏
01-09 1084
【代码】tritonserver学习之三：tritonserver源码编译。
TensorRT部署-使用TensorRT+Triton加速部署YOLOv4-项目实战-附完整流程教程.zip
01-18
4. **模型部署**：将优化后的TensorRT模型上传到Triton的模型仓库，定义模型配置文件，说明模型输入输出、数据类型和形状。 5. **客户端接口开发**：编写客户端代码，发送推理请求，并接收和解析响应。可以使用各种...
算法部署-基于Triton部署YOLO目标检测算法-支持YOLOv9-附项目源码+流程教程-优质项目实战.zip
最新发布
07-20
算法部署_基于Triton部署YOLO目标检测算法_支持YOLOv9_附项目源码+流程教程_优质项目实战
大模型部署-使用Triton+TensorRT-LLM部署ChatGLM3-6B大模型-附项目源码+流程教程-优质项目实战
07-07
大模型部署_使用Triton+TensorRT-LLM部署ChatGLM3-6B大模型_附项目源码+流程教程_优质项目实战
TensorRT-tensorrt的triton后端-backend.zip
02-28
当用户将基于TensorRT优化的模型部署到Triton服务器时，Triton会调用TensorRT的API来执行模型推理，从而实现高效的数据处理和计算。 在"TensorRT_tensorrt的triton后端_backend.zip"这个压缩包中，可能包含了与配置...
tritonserver的metrics参数说明
wentinghappyday的博客
11-22 1502
tritonserver 自带的小工具 metrics使用指南，实现服务性能评估以及配置最佳实例数量的方式
tritonserver学习之一：triton使用流程
liupenglove的专栏
12-27 1663
进入client镜像，测试tritonserver推理。
Triton_server部署学习笔记
dream_home8407的博客
07-17 695
下载镜像。
Triton Inference Server 环境配置
qq_26815239的博客
11-28 1390
Triton Inference Server 环境配置，Ubuntu18.04，3090显卡，显卡驱动版本510.85.02，cuda版本11.6，docker版本20.10.12
tritonserver学习之五：backend实现机制
liupenglove的专栏
01-24 1652
TRITONBACKEND_Initialize：创建ModelState对象。TRITONBACKEND_ModelInstanceInitialize：创建ModelInstanceState对象，可以通过这个对象调用LoadModel函数tritonserver学习之二：tritonserver编译-CSDN博客，博客中介绍的backend的类，主要还是一个proxy的作用，直接调用深度学习api的还是对应的backend，大家可以自己学习理解下，不清楚的，欢迎评论交流。
Pytorch实现RNN预测模型并使用C++相应的ONNX模型推理
午夜的行人
09-15 621
Pytorch实现RNN预测模型并使用C++相应的ONNX模型推理。
Triton算法服务部署：初识与试用【Hello world】
AI数据工厂
12-13 412
介绍了Triton，并结合官方文档给出了一个Hello World级别的使用示例。
triton server教程
06-13
Triton Server是一个用于部署机器学习模型的开源平台，支持多种框架（如TensorFlow、PyTorch、ONNX等）和多种部署方式（如RESTful API、gRPC等）。以下是Triton Server的安装和使用教程： 1. 安装NVIDIA CUDA和cuDNN Triton Server依赖于NVIDIA CUDA和cuDNN，因此需要先安装它们。具体安装步骤可参考官方文档：https://docs.nvidia.com/cuda/index.html 和 https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html 。 2. 下载Triton Server 可以从Triton Server的GitHub仓库下载最新版本：https://github.com/triton-inference-server/server 。 3. 安装Triton Server Triton Server的安装非常简单，只需解压缩下载的压缩包即可。 ``` $ tar xzf tritonserver-2.6.0-linux-x86_64.tar.gz ``` 4. 配置模型 Triton Server通过配置文件来加载和部署模型。可以参考官方文档中的模型配置示例（https://github.com/triton-inference-server/server/tree/main/docs/examples/model_repository ）来创建自己的配置文件。配置文件将指定模型的路径、输入和输出张量的名称和形状、部署方式等信息。 5. 启动Triton Server 启动Triton Server非常简单，只需运行以下命令即可： ``` $ ./tritonserver-2.6.0/bin/tritonserver --model-repository=/path/to/model/repository ``` 其中，--model-repository参数指定了存放模型的目录。 6. 使用Triton Server 启动Triton Server后，可以通过RESTful API或gRPC接口来部署和查询模型。具体使用方法可参考官方文档：https://github.com/triton-inference-server/server 。 希望这些步骤能够帮助您安装和使用Triton Server。
评论 3
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值