Win11系统C++将ONNX转换为engineer文件,TensorRT加速推理

已于 2022-10-30 22:28:51 修改
阅读量3k 收藏 26
点赞数 6
分类专栏： C++ CUDA pytorch 文章标签： c++ onnx
于 2022-10-30 22:25:06 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/Vertira/article/details/127602349
版权
pytorch 同时被 3 个专栏收录
86 篇文章 20 订阅
订阅专栏
CUDA
14 篇文章 2 订阅
订阅专栏
C++
13 篇文章 0 订阅
订阅专栏

准确工作，安装配置好CUDA,cudnn，vs2019，TensorRT

可以参考我博客（下面博客有CUDA11.2,CUDNN11.2，vs2019，TensorRT配置方法）

(70条消息) WIN11+CUAD11.2+vs2019+tensorTR8.6+Yolov3/4/5模型加速_Vertira的博客-CSDN博客

下面我们在上面的基础上，下载opencv4（备用），然后创建onnx2TensorRT 项目

1.vs2019创建控制台程序，

  如果你是初次安装，没有c++套件（如果你安装了C++套件 ，忽略此步，直接进行第2步），

打开 Visual Studio Installer , 点击 " 修改 " 按钮 

进入 可以安装套件的界面

安装 " 使用 C++ 的桌面开发 " 组件 ; 有很多组件，这里咱们只安装C++

 选中后 , 右下角会显示 " 修改 " 按钮 , 点击该按钮 , 即可开始

 

然后等待安装完成即可 ;（时间有点长）

然后：

2、创建并运行 Windows 控制台程序 

 文件-->新建---》项目

选择创建 " 控制台应用 " ,

下一步，

在弹出的界面中 定义好工程名称onnx2TensorRT  和路径，确定创建  即可 

 然后进入工程文件夹，打开工程，工程自动创建带有输出hello world的cpp源文件，把文件清空

然后粘贴下面的代码：（由于没有配置路径，会出现很多红线，这个不用担心，下面就开始配置CUDA,TensorRT,这里没有用的图片显示，我暂时没有配置opencv）

#include <iostream>
#include <fstream>
#include "NvInfer.h"
#include "NvOnnxParser.h"

// 实例化记录器界面。捕获所有警告消息，但忽略信息性消息
class Logger : public nvinfer1::ILogger
{
    void log(Severity severity, const char* msg) noexcept override
    {
        // suppress info-level messages
        if (severity <= Severity::kWARNING)
            std::cout << msg << std::endl;
    }
} logger;


void ONNX2TensorRT(const char* ONNX_file, std::string save_ngine)
{
    // 1.创建构建器的实例
    nvinfer1::IBuilder* builder = nvinfer1::createInferBuilder(logger);

    // 2.创建网络定义
    uint32_t flag = 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
    nvinfer1::INetworkDefinition* network = builder->createNetworkV2(flag);

    // 3.创建一个 ONNX 解析器来填充网络
    nvonnxparser::IParser* parser = nvonnxparser::createParser(*network, logger);

    // 4.读取模型文件并处理任何错误
    parser->parseFromFile(ONNX_file, static_cast<int32_t>(nvinfer1::ILogger::Severity::kWARNING));
    for (int32_t i = 0; i < parser->getNbErrors(); ++i)
    {
        std::cout << parser->getError(i)->desc() << std::endl;
    }

    // 5.创建一个构建配置，指定 TensorRT 应该如何优化模型
    nvinfer1::IBuilderConfig* config = builder->createBuilderConfig();

    // 6.设置属性来控制 TensorRT 如何优化网络
    // 设置内存池的空间
    config->setMemoryPoolLimit(nvinfer1::MemoryPoolType::kWORKSPACE, 16 * (1 << 20));
    // 设置低精度   注释掉为FP32
    if (builder->platformHasFastFp16())
    {
        config->setFlag(nvinfer1::BuilderFlag::kFP16);
    }

    // 7.指定配置后，构建引擎
    nvinfer1::IHostMemory* serializedModel = builder->buildSerializedNetwork(*network, *config);

    // 8.保存TensorRT模型
    std::ofstream p(save_ngine, std::ios::binary);
    p.write(reinterpret_cast<const char*>(serializedModel->data()), serializedModel->size());

    // 9.序列化引擎包含权重的必要副本，因此不再需要解析器、网络定义、构建器配置和构建器，可以安全地删除
    delete parser;
    delete network;
    delete config;
    delete builder;

    // 10.将引擎保存到磁盘，并且可以删除它被序列化到的缓冲区
    delete serializedModel;
}


void exportONNX(const char* ONNX_file, std::string save_ngine)
{
    std::ifstream file(ONNX_file, std::ios::binary);
    if (!file.good())
    {
        std::cout << "Load ONNX file failed! No file found from:" << ONNX_file << std::endl;
        return ;
    }

    std::cout << "Load ONNX file from: " << ONNX_file << std::endl;
    std::cout << "Starting export ..." << std::endl;

    ONNX2TensorRT(ONNX_file, save_ngine);

    std::cout << "Export success, saved as: " << save_ngine << std::endl;

}


int main(int argc, char** argv)
{
    // 输入信息
    const char* ONNX_file  = "../weights/test.onnx";
    std::string save_ngine = "../weights/test.engine";

    exportONNX(ONNX_file, save_ngine);

    return 0;
}

 

 然后，开始配置

属性界面如下： 

 包含目录：是CUDA的include文件夹所在路径和TensorRT的 include文件夹

$(CUDA_PATH)\include                            
D:\software\TensorRT-8.4.1.5_CUDA11.6_Cudnn8.4.1\include
D:\software\TensorRT-8.4.1.5_CUDA11.6_Cudnn8.4.1\samples\common


库目录：指的是 CUDA的lib 和TensorRT的lib

$(CUDA_PATH)\lib
$(CUDA_PATH)\lib\x64
D:\software\TensorRT-8.4.1.5_CUDA11.6_Cudnn8.4.1\lib

注意：$(CUDA_PATH)  也可以用绝对路径代替

3. 设置属性 — 链接器 

 输入的内容：

nvinfer.lib
nvinfer_plugin.lib
nvonnxparser.lib
nvparsers.lib
cudnn.lib
cublas.lib
cudart.lib

然后确定，确定  即可

5. 创建模型文件夹

新建weights文件夹，将你的onnx文件放在里面

      

 注意：weights路径的位置一定要和程序  最下面的路径位置一样

 然后，在debug模型在运行，如果你需要release版本，那就在release模式下运行。

 运行结果：

Load ONNX file from: ../weights/model.onnx
Starting export ...
onnx2trt_utils.cpp:369: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
TensorRT was linked against cuDNN 8.4.1 but loaded cuDNN 8.2.1
Weights [name=Conv_3 + Relu_4.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Conv_3 + Relu_4.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Conv_3 + Relu_4.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Conv_3 + Relu_4.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Conv_3 + Relu_4.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Conv_3 + Relu_4.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Conv_3 + Relu_4.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Conv_3 + Relu_4.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Conv_3 + Relu_4.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
 - Values less than smallest positive FP16 Subnormal value detected. Converting to FP16 minimum subnormalized value.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_8 + Relu_9.bias] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
Weights [name=Gemm_10.weight] had the following issues when converted to FP16:
 - Subnormal FP16 values detected.
If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights.
The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
Export success, saved as: ../weights/model.engine

D:\VS_project\onnx2TensorRT\x64\Debug\onnx2TensorRT.exe (进程 17936)已退出，代码为 0。
要在调试停止时自动关闭控制台，请启用“工具”->“选项”->“调试”->“调试停止时自动关闭控制台”。
按任意键关闭此窗口. . .

从最后两句话可以看出

The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
Export success, saved as: ../weights/model.engine

D:\VS_project\onnx2TensorRT\x64\Debug\onnx2TensorRT.exe (进程 17936)已退出，代码为 0。
要在调试停止时自动关闭控制台，请启用“工具”->“选项”->“调试”->“调试停止时自动关闭控制台”。
按任意键关闭此窗口. . .

程序已经运行成功，没有报错。

7.注意：如果运行时出现 如下警告

问题 — 缺少 zlibwapi.dll 文件

若出现缺少zlibwapi.dll，需要下载，

若出现缺少zlibwapi.dll，需要下载，
链接：https://pan.baidu.com/s/12sVdiDH-NOOZNI9QqJoZuA?pwd=a0n0

 链接：百度网盘 请输入提取码百度网盘为您提供文件的网络备份、同步和分享服务。空间大、速度快、安全稳固，支持教育网加速，支持手机端。注册使用百度网盘即可享受免费存储空间https://pan.baidu.com/s/12sVdiDH-NOOZNI9QqJoZuA?pwd=a0n0
提取码：a0n0
下载后解压，将zlibwapi.dll 放入 …\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin(我的路径)
重新运行程序。如果没有错误，就忽略这个注意。

8. 生成engine文件

选择Release模式，然后重新配置一遍

点击，更换为Release

结束

欢迎   点赞   收藏   加   关注

参考：

(56条消息) Win10系统C++将ONNX转换为TensorRT_田小草呀的博客-CSDN博客_c++ onnx转tensorrt

Vertira
关注
  • 6
    点赞
  • 踩
  • 26
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
专栏目录
tensorrt c++ 推理项目
07-03
tensorrt c++ 推理示例项目，支持分类网络，比如alexnet...转换流程：pytorch转onnx，onnx转tensorrt引擎 python生成tensorrt引擎步骤，可以查看我的博客： https://blog.csdn.net/jacke121/article/details/125382721
yolov7 的win10 tensorrt推理 c++ 版本
01-27
tensorrtx/yolov7 at master · wang-xinyu/tensorrtx · GitHub yolov7的win10 cuda11.4.3 cudnn8.2 tensorrt8.2.1.8 推理
c++通过tensorRT调用模型进行推理
qq_26611129的博客
09-07 2200
拿到onnx模型后，通过tensorRT将onnx模型转换为对应的engine模型，并且在bin目录下生成.engine模型文件。算法工程师训练得到的onnx模型。
onnx模型转engine并进行推理全过程解析
AI小花猫的博客
03-24 4803
TensorRT C++ API 使用ONNX进行模型构建和推理的全过程
3、TensorRT学习笔记之ONNX转engine
NOZhengYuan的博客
04-25 1220
主要讲解了ONNX转TensorRT流程、代码。
OnnxRuntime TensorRT OpenCV::DNN性能对比(YoloV8)实测
qq_18236971的博客
07-28 3520
cpu选择onnxruntime或者dnn都可以,建议选择ort.gpu选择tensorrt,如果有兼容需求就只能选择onnxruntime了.不得不说,gpu推理上TRT把ORT薄纱了,不需要warm-up,对工业生产环境非常友好,因为在实际生产环境中,都不是实时推理,而是有间隔的推理,ORT在一段间隔时间后cuda性能会有所衰减,当然也可能是我还没摸透ort这个框架,欢迎大佬指正.
【TensorRT】onnx转成TensorRT开发踩坑记录（Win10+cuda11.1+TensorRT7.2.2.3）（持续更新）
qq_42944019的博客
12-16 5038
本博客提供onnx模型转成TensorRT模型时的开发经验。
从零开始 TensorRT（7）C++ 篇：解析 ONNX
秋山丶雪绪的博客
03-07 853
但是反复检查代码感觉没有非法的内存访问，偶然对常用输入的 BatchSize 修改后发现代码能跑通，便做了如下测试。按理来说 optBatchSize 和校正时的 BatchSize 没什么关系，多半是 TensorRT 内部的 Bug。将读取 Numpy 文件的部分改为读取本地图像，并利用 OpenCV 对图像进行预处理，Int8 模式中的校正器部分代码也有所改动。文件，并在 C++ 部分直接读取 Numpy 文件中的数据用于推理和校正，避免了图片解码的相关代码。在 cookbook 中，
yolov5.zip c++推理yolov5 onnx模型
01-02
yolov5 pytorch模型转onnx并用python ,c++ ，java进行推理
使用C++部署RTMPose的onnx模型或者tensorrt模型源码+使用说明.zip
最新发布
05-15
RTMPose-Deploy是一个不使用mmdeploy进行RTMPose本地化部署C++示例。 目前我们在Windows系统上使用OnnxRuntime CPU SDK对...后续会考虑添加在Windows系统上使用C++ TensorRT SDK对RTMDetnano和RTMPose进行部署。
C++使用onnxruntime/opencv对onnx模型进行推理(附代码)
05-23
C++使用onnxruntime/opencv对onnx模型进行推理(附代码)
深度学习项目部署：使用TensorRT通过ONNX部署Pytorch项目
HollowKnightz的博客
12-17 4829
TensorRT部署Pytorch项目
【安装】win下的openvino安装及onnx模型转换.xml和.bin
kk的博客
08-15 2139
openvino的安装及.onnx模型转.xml及.bin
TensorRT onnx转engine报Assertion failed: dims.nbDims == 4 || dims.nbDims == 5
修炼之路
08-27 2195
错误信息 在使用TensorRT将onnx转为engine的时候报错，错误信息如下 [08/27/2021-15:27:08] [I] === Model Options === [08/27/2021-15:27:08] [I] Format: ONNX [08/27/2021-15:27:08] [I] Model: glintr100.onnx [08/27/2021-15:27:08] [I] Output: [08/27/2021-15:27:08] [I] === Build Options =
onnx模型转TensorRT模型时出错
weixin_48192326的博客
06-08 1117
【代码】把onnx模型转TensorRT模型出错。
python .onnx 转 .engine亲测ok
4月16！的博客
11-30 1033
python版本 onnx文件转engine
基于RESNET网络实现tensorrt转换教程(onnx转engine和wts转engine)
weixin_38252409的博客
08-29 591
近日很想验证使用pytorch训练模型转tensorrt各种关系，更深理解基于C++ API接口engine加速理论(Python API接口稍微简单，将不在验证)，本文基于ResNet分类网络。 本文内容主要分为六个内容，第一个内容介绍使用python构建网络，获取pt/wts/onnx文件；第二个内容介绍基于C++ API构建engine；第三个内容介绍基于C++使用onnx构建 engin...
TensorRT快速上手指南
pytorch内部机制解读
09-09 1619
最近一段时间接触了TensorRT，因此顺带整理了一份TensorRT中一些常用关键组件的笔记，以方便未来能够快速查阅和上手。需要注意的是本文只是一些常用特性与关键组件的梳理，而不是一个小白入门的教程，所以后续文章内容并不会提供快速上手的样例代码，实际上TensorRT官方的example中已经挺完善了，从0到1入门的新手建议直接看官方example中的代码。
onnx模型转engine模型
wzy_777的博客
02-26 479
onnx模型转engine模型
如何将自己训练好的pytroch模型转换成c++ 的tensorRT推理模型
04-05
要将PyTorch模型转换为TensorRT推理模型，需要执行以下步骤： 1. 安装TensorRT：在NVIDIA的官方网站上下载适用于您的系统的TensorRT，然后按照说明进行安装。 2. 将PyTorch模型转换为ONNX格式：使用PyTorch的“torch.onnx.export”函数将PyTorch模型转换为ONNX格式。确保指定输入和输出张量的形状和数据类型，以便在TensorRT中正确解释模型。 3. 使用TensorRT的“uff.from_onnx”函数将ONNX模型转换为UFF格式。 UFF是一种高效的模型格式，可在TensorRT中实现更快的推理性能。 4. 使用TensorRT的“create_inference_engine”函数将UFF模型转换为TensorRT推理引擎。此函数将创建TensorRT的推理引擎对象，该对象将用于推理过程。 5. 加载数据和运行推理：加载测试数据并将其输入到推理引擎中，然后将输出与预期输出进行比较以验证模型的正确性。 以下是一个示例代码片段，演示了如何将PyTorch模型转换为TensorRT推理模型： ```Python import torch import tensorrt as trt import numpy as np # Step 1: Install TensorRT and verify the installation # Step 2: Convert PyTorch model to ONNX format pytorch_model = ... # your PyTorch model dummy_input = torch.randn(...) # create a dummy input tensor input_names = ... # list of input tensor names output_names = ... # list of output tensor names onnx_model_path = "model.onnx" # path to save the ONNX model torch.onnx.export(pytorch_model, dummy_input, onnx_model_path, input_names=input_names, output_names=output_names) # Step 3: Convert ONNX model to UFF format uff_model_path = "model.uff" # path to save the UFF model uff_model = trt.UffParser().parse(onnx_model_path, output_names) uff_model.write_to_file(uff_model_path) # Step 4: Create TensorRT inference engine TRT_LOGGER = trt.Logger(trt.Logger.WARNING) trt_runtime = trt.Runtime(TRT_LOGGER) engine_path = "model.engine" # path to save the TensorRT engine with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.UffParser() as parser: builder.max_workspace_size = 1 << 30 # set maximum workspace size builder.max_batch_size = 1 # set maximum batch size builder.fp16_mode = True # enable FP16 precision input_shape = (..., ...) # set input shape input_tensor = network.add_input(input_names[0], trt.float32, input_shape) output_tensor = parser.parse_buffer(output_names[0], uff_model) network.mark_output(output_tensor) engine = builder.build_cuda_engine(network) with open(engine_path, "wb") as f: f.write(engine.serialize()) # Step 5: Load data and run inference input_data = np.random.random(input_shape).astype(np.float32) with engine.create_execution_context() as context: output_data = np.empty_like(output_tensor.host_buffer) context.execute(batch_size=1, bindings=[int(input_tensor.device_buffer), int(output_tensor.device_buffer)]) np.copyto(output_data, output_tensor.host_buffer) expected_output_data = ... # expected output data assert np.allclose(output_data, expected_output_data, rtol=1e-3, atol=1e-3) # verify output ```

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值