tensorRT插件

tensorRT自定义算子插件

enqueue
实际插件op的执行函数，我们自己实现的cuda操作就放到这里
https://oldpan.me/archives/tensorrt-plugin-one-post-get-it

TensorRT插件的存在目的，主要是为了让我们实现TensorRT目前还不支持的算子
https://zhuanlan.zhihu.com/p/297002406

 
https://github.com/zerollzeng/tiny-tensorrt/blob/master/docs/CustomPlugin-CN.md
例子 
https://github.com/zerollzeng/tiny-tensorrt/blob/master/plugin/PReLUPlugin/PReLUPlugin.cu

 
https://docs.nvidia.com/deeplearning/tensorrt/api/c_api/classnvinfer1_1_1_dims.html

c++的torch::Tensor操作

c++ 部署libtorch 时对Tensor块的常用操作API (例如vector转换为torch、squeeze、select、select_indxe、max等）
https://www.codenong.com/cs105839096/

at::Tensor pcl_losses_forward_cuda(const at::Tensor& pcl_probs,
    const at::Tensor& labels,
    const at::Tensor& cls_loss_weights,
    const at::Tensor& pc_labels,
    const at::Tensor& pc_probs,
    const at::Tensor& img_cls_loss_weights,
    const at::Tensor& im_labels) {
    int batch_size = pcl_probs.size(0);
    int channels = pcl_probs.size(1);
    int num_positive = pc_labels.size(1);
    at::Tensor output = at::empty({1, channels}, pcl_probs.options());
    cudaStream_t stream = at::cuda::getCurrentCUDAStream();
    PCLLossesForwardLaucher(
        pcl_probs.data<float>(),
        labels.data<float>(),
        cls_loss_weights.data<float>(),
        pc_labels.data<float>(),
        pc_probs.data<float>(),
        img_cls_loss_weights.data<float>(),
        im_labels.data<float>(),
        batch_size, channels, num_positive,
        output.data<float>(),
        stream
    );
    return output;
}
https://dongsky.github.io/2020/04/04/pytorch1xcppext/
 

PluginV2Layer must be V2DynamicExt when there are runtime input dimensions  Assertion failed: axis >= 0 && axis < nbDims

I have solved this Issue by Inheriting IPluginv2DynamicExt Class instead of IPluginv2Ext Class.
https://forums.developer.nvidia.com/t/pluginv2layer-must-be-v2dynamicext-when-there-are-runtime-input-dimensions/168992/5

Yes I believe you'll have to tweak the existing implementation to implement the IPluginV2DynamicExt interface instead. Please try this, and feel free to submit a pull request if you get a working implementation.
https://github.com/NVIDIA/TensorRT/issues/544

The error indicates your model using a dynamic shape, but the plugin only supports the static version.
Please provide a batch size parameter when generating the onnx model to get a static version.
https://forums.developer.nvidia.com/t/yolov4-tensorrt-pluginv2layer-must-be-v2dynamicext-when-there-are-runtime/161123

You should use the BatchedNMSDynamic_TRT plugin if you have dynamic shapes. This plugin is supported in TensorRT version > 7.2.
https://github.com/NVIDIA/TensorRT/issues/795

DimsExprs getOutputDimensions
https://forums.developer.nvidia.com/t/getoutputdimensions-for-ipluginv2dynamicext/149429
https://docs.nvidia.com/deeplearning/tensorrt/api/c_api/classnvinfer1_1_1_i_plugin_v2_dynamic_ext.html#a3d19937b5f81272b29ee7c986ca297ed

// First dimension of output is sum of input 
        // first dimensions.
        DimsExprs output(inputs[0]);
        output.d[0] = 
            exprBuilder.operation(DimensionOperation::kSUM, 
                inputs[0].d[0], inputs[1].d[0]);
https://www.codetd.com/article/12453558
 

I think that you should read first sentences of this section: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#ptx-compatibility . NVIDIA GPUs have a notion of compute compatibility and some functions are not enabled by compiler by default unless you specify that you compile on a device with certain compute compatibility. For example, on my server, Titan X Tesla has compute compatibility 6.1, so I compiled it with compute_61 arch.

I think you should also refer to this table. https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html#hardware-precision-matrix  . You can find out immediately if the GPU on your server supports Fast FP16. For example, as I mentioned that Titan X (pascal) has compute capability 6.1 and the table indicates that it doesn't support fast fp16.