TensorRT实现yolov5推理加速（二）

一、参考资料

TensorRT实现yolov5推理加速（一）
yolov5_tensorrt_int8
tensorrtx
TensorRT int8 量化部署 yolov5s 5.0 模型
Tensorrt环境安装及yolov5模型转换以及量化部署INT8

二、重要说明

1. github源码: tensorrtx

三、可能出现的问题

• dnn未安装

/home/yichao/MyDocuments/tensorrtx/yolov5/calibrator.cpp:4:10: fatal error: opencv2/dnn/dnn.hpp: 没有那个文件或目录
 #include <opencv2/dnn/dnn.hpp>
          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
CMakeFiles/yolov5.dir/build.make:75: recipe for target 'CMakeFiles/yolov5.dir/calibrator.cpp.o' failed
make[2]: *** [CMakeFiles/yolov5.dir/calibrator.cpp.o] Error 1
CMakeFiles/Makefile2:84: recipe for target 'CMakeFiles/yolov5.dir/all' failed
make[1]: *** [CMakeFiles/yolov5.dir/all] Error 2
Makefile:90: recipe for target 'all' failed
make: *** [all] Error 2

/home/yichao/MyDocuments/tensorrtx/yolov5/calibrator.cpp:4:31: fatal error: opencv2/dnn/dnn.hpp: 没有那个文件或目录
 #include <opencv2/dnn/dnn.hpp>
                               ^
compilation terminated.
CMakeFiles/yolov5.dir/build.make:62: recipe for target 'CMakeFiles/yolov5.dir/calibrator.cpp.o' failed
make[2]: *** [CMakeFiles/yolov5.dir/calibrator.cpp.o] Error 1
make[2]: *** 正在等待未完成的任务....
In file included from /home/yichao/MyDocuments/tensorrtx/yolov5/common.hpp:8:0,
                 from /home/yichao/MyDocuments/tensorrtx/yolov5/yolov5.cpp:6:
/usr/include/opencv2/opencv.hpp:83:29: fatal error: opencv2/shape.hpp: 没有那个文件或目录
compilation terminated.
CMakeFiles/yolov5.dir/build.make:86: recipe for target 'CMakeFiles/yolov5.dir/yolov5.cpp.o' failed
make[2]: *** [CMakeFiles/yolov5.dir/yolov5.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/yolov5.dir/all' failed
make[1]: *** [CMakeFiles/yolov5.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

[libopencv-dnn-dev_4.2.0+dfsg-5_arm64.deb](https://ubuntu.pkgs.org/20.04/ubuntu-universe-arm64/libopencv-dnn-dev_4.2.0+dfsg-5_arm64.deb.html)

[cannot make reinaface](https://github.com/wang-xinyu/tensorrtx/issues/373)

错误原因：
没有安装libopencv-dnn-dev

解决办法：
安装libopencv-dnn-dev
如果已经安装opencv，libopencv-dnn-dev 会自动匹配对应的opencv版本

sudo apt-get update
sudo apt-get install libopencv-dnn-dev

yichao@yichao:/usr/include$ sudo apt-get install libopencv-dnn-dev
正在读取软件包列表... 完成
正在分析软件包的依赖关系树       
正在读取状态信息... 完成       
将会同时安装下列软件：
  libopencv-core-dev libopencv-core3.3 libopencv-dnn3.3 libopencv-imgproc3.3
下列【新】软件包将被安装：
  libopencv-core-dev libopencv-core3.3 libopencv-dnn-dev libopencv-dnn3.3 libopencv-imgproc3.3
升级了 0 个软件包，新安装了 5 个软件包，要卸载 0 个软件包，有 22 个软件包未被升级。
需要下载 988 kB/5,124 kB 的归档。
解压缩后会消耗 26.8 MB 的额外空间。
您希望继续执行吗？ [Y/n] y

# 查看是否安装 libopencv-dnn-dev
dpkg -l | grep dnn

ii  libcudnn7                                                    7.6.5.32-1+cuda10.0                             amd64        cuDNN runtime libraries
ii  libcudnn7-dev                                                7.6.5.32-1+cuda10.0                             amd64        cuDNN development libraries and headers
ii  libcudnn7-doc                                                7.6.5.32-1+cuda10.0                             amd64        cuDNN documents and samples

• 缺少文件

In file included from /home/yichao/MyDocuments/tensorrtx/yolov5/common.hpp:8:0,
                 from /home/yichao/MyDocuments/tensorrtx/yolov5/yolov5.cpp:6:
/usr/include/opencv2/opencv.hpp:56:10: fatal error: opencv2/calib3d.hpp: 没有那个文件或目录
 #include "opencv2/calib3d.hpp"
          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.

opencv2/features2d.hpp: 没有那个文件或目录
 #include "opencv2/features2d.hpp"
          ^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

opencv2/flann/miniflann.hpp: 没有那个文件或目录
 #include "opencv2/flann/miniflann.hpp"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~

解决办法：
1. 找到calib3d.hpp在哪里
/usr/local/opencv3.3.0/include/opencv2/calib3d.hpp 

2. 创建软链接
sudo ln -s /usr/local/opencv3.3.0/include/opencv2/calib3d.hpp /usr/include/opencv2/calib3d.hpp

如果缺少的文件太多了，可能是编译opencv路径错误，尝试用终极方法：

# 创建软链接
sudo ln -s /usr/local/opencv3.3.0/include/opencv2 /usr/include/opencv2

• 源代码错误

/home/yichao/MyDocuments/tensorrtx/yolov5/calibrator.cpp: In member function ‘virtual bool Int8EntropyCalibrator2::getBatch(void**, const char**, int)’:
/home/yichao/MyDocuments/tensorrtx/yolov5/calibrator.cpp:52:132: error: too many arguments to function ‘cv::Mat cv::dnn::experimental_dnn_v1::blobFromImages(const std::vector<cv::Mat>&, double, cv::Size, const Scalar&, bool)’
 cv::Mat blob = cv::dnn::blobFromImages(input_imgs_, 1.0 / 255.0, cv::Size(input_w_, input_h_), cv::Scalar(0, 0, 0), true, false);

错误原因：
源码错误
/home/yichao/MyDocuments/tensorrtx/yolov5/calibrator.cpp:52

解决办法：
修改源码

cv::Mat blob = cv::dnn::blobFromImages(input_imgs_, 1.0 / 255.0, cv::Size(input_w_, input_h_), cv::Scalar(0, 0, 0), true, false);
修改为
cv::Mat blob = cv::dnn::blobFromImages(input_imgs_, 1.0 / 255.0, cv::Size(input_w_, input_h_), cv::Scalar(0, 0, 0), true);

• TensorRT错误

/usr/include/opencv2/core/cvstd.hpp:647：对‘cv::String::deallocate()’未定义的引用
CMakeFiles/yolov5.dir/yolov5.cpp.o：在函数‘main’中：
/usr/include/opencv2/core/cvstd.hpp:647：对‘cv::String::deallocate()’未定义的引用
/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/libnvinfer.so：对‘cublasCreate_v2@libcublas.so.10.0’未定义的引用
/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/libnvinfer.so：对‘cublasSgemmEx@libcublas.so.10.0’未定义的引用

方法一：

错误原因：
找不到opencv包

解决办法：
修改文件
/home/yichao/MyDocuments/tensorrtx/yolov5/CMakeLists.txt

# 取消注释
find_package(OpenCV)
include_directories(${OpenCV_INCLUDE_DIRS})

方法二：

如果方法一不行，尝试方法二

错误原因：
博主编译 opencv3.3.0 使用的cuda版本是 10.2
而之前安装的 tensorRT7.1.3 支持的cuda是 10.0
cuda版本不匹配，导致yolov5 make失败

# 查看 tensorRT 的版本
dpkg -l | grep TensorRT

解决办法：
卸载 tensorRT，重装 tensorRT

# 卸载 tensorRT
sudo apt-get purge "libnvinfer*"
sudo apt-get autoremove

修改文件
/home/yichao/MyDocuments/tensorrtx/yolov5/CMakeLists.txt

include_directories(/usr/include/x86_64-linux-gnu/)
link_directories(/usr/lib/x86_64-linux-gnu/)
修改为
include_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/include/)
link_directories(/home/yichao/360Downloads/TensorRT-7.1.3.4/lib/)

• tensorRT版本不匹配

[TensorRT] ERROR: ../rtSafe/coreReadArchive.cpp (38) - Serialization Error in verifyHeader: 0 (Version tag does not match)
[TensorRT] ERROR: INVALID_STATE: std::exception
[TensorRT] ERROR: INVALID_CONFIG: Deserialize the cuda engine failed.
Traceback (most recent call last):
  File "yolov5_trt.py", line 374, in <module>
    yolov5_wrapper = YoLov5TRT(engine_file_path)
  File "yolov5_trt.py", line 89, in __init__
    context = engine.create_execution_context()
AttributeError: 'NoneType' object has no attribute 'create_execution_context'
-------------------------------------------------------------------
PyCUDA ERROR: The context stack was not empty upon module cleanup.
-------------------------------------------------------------------
A context was still active when the context stack was being
cleaned up. At this point in our execution, CUDA may already
have been deinitialized, so there is no way we can finish
cleanly. The program will be aborted now.
Use Context.pop() to avoid this problem.
-------------------------------------------------------------------
已放弃 (核心已转储)

错误原因一：
[TensorRT] ERROR: ../rtSafe/coreReadArchive.cpp (38) - Serialization Error in verifyHeader: 0 (Version tag does not match)

[](https://forums.developer.nvidia.com/t/tensorrt-error-rtsafe-corereadarchive-cpp-31-serialization-error-in-verifyheader-0-magic-tag-does-not-match/81872)

If the engine was created and ran on different versions, this may happen. TensorRT engines are not compatible across different TensorRT versions.

cuda版本与tensorRT版本不匹配
博主使用c++编译生成的yolov5s.engine，tensorRT版本是7.1.3，对应的cuda版本是10.2；
在python API中trt推理，tensorRT版本是7.1.3，对应的cuda版本是10.0。

解决办法：
卸载tensorRT，重新安装
pip安装python API的tensorRT，自动寻找系统的cuda版本，然后编译安装对应的tensorRT链接库。所以只需要pip卸载tensorRT，重新安装即可，使用之前的tensorRT安装包即可。

• yolov5 INT8量化后不出框

mAP of the INT8 TensorRT engine of the "yolov4-608" model is not good.

[yolov5官方github](https://github.com/ultralytics/yolov5)
[Quantization Error Analysis on Object Detection](https://github.com/NVIDIA/TensorRT/issues/1114)
[](https://github.com/NVIDIA/TensorRT/issues/997)

错误原因：
yolov5/models/common.py

王宇鑫解释：
yolov5默认的激活函数是 nn.SiLU()
tensorRT支持SiLU激活函数，但不一定量化效果就好，可能现在量化依然只对ReLU激活函数有较好的效果。

陈锐解释：
FP16量化其实挺快的，int8没比fp16快多少。

sky_hole 解释：
FP16量化足够了，不要用INT8。
在一些比较大的模型上，INT8提升速度显著。
在一些只关注某种衡量指标（比如，相似度等）的模型上，INT8提升速度显著。

ttyio 解释：
we see it works for networks like quartzNet, efficientNet.


方法一：
nn.SiLU()修改为nn.LeakyReLU()，重新训练

方法二（待验证）：
使用低版本的tensorRT 7.0.0