小小甜菜tensorrt爬坑记

小小甜菜tensorrt爬坑记

遍观全球，模型优化部署方案中生产化最好的是intel，英伟达，谷歌（TPU）。我在研究了intel的openvino后，有感于CPU和VPU的缺陷，准备使用tensorrt作为补充。

安装

sudo apt-get install nvidia-driver-418
sudo chmod +x cuda_10.0.130_410.48_linux.run
sudo ./cuda_10.0.130_410.48_linux.run
sudo gedit ~/.bashrc
export CUDA_HOME=/usr/local/cuda 
export PATH=$PATH:$CUDA_HOME/bin 
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
source ~/.bashrc
tar -xzvf cudnn-10.0-linux-x64-v7.6.3.30.tgz 
sudo cp cuda/include/cudnn.h /usr/local/cuda/include/ 
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64/ 
sudo chmod a+r /usr/local/cuda/include/cudnn.h 
sudo chmod a+r /usr/local/cuda/lib64/libcudnn*
sudo apt install python3-pip
pip3 install numpy  -i https://pypi.tuna.tsinghua.edu.cn/simple
sudo apt-get install python3.6-dev
pip3 install pycuda  -i https://pypi.tuna.tsinghua.edu.cn/simple 
//官网下载对于版本
tar xzvf TensorRT-7.0.0.11.Ubuntu-18.04.x86_64-gnu.cuda-10.0.cudnn7.6.tar.gz 
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/xs/TensorRT-7.0.0.11/lib
cd TensorRT-7.0.0.11/python
sudo pip3 install tensorrt-7.0.0.11-cp36-none-linux_x86_64.whl
cd ../uff
sudo pip3 install uff-0.6.5-py2.py3-none-any.whl -i https://pypi.tuna.tsinghua.edu.cn/simple
cd ../graphsurgeon
sudo pip3 install graphsurgeon-0.4.1-py2.py3-none-any.whl
pip3 install tensorflow-gpu  -i https://pypi.tuna.tsinghua.edu.cn/simple 
cd ../lib
sudo cp libnvinfer.so.7 /usr/lib/
sudo cp libnvonnxparser.so.7 /usr/lib/
sudo cp libmyelin.so.1 /usr/lib/
sudo cp libnvparsers.so.7 /usr/lib/
sudo cp libnvinfer_plugin.so.7 /usr/lib/

ONNX框架

使用tensorrt就不得不提到微软和facebook合力创建的开源社区平台ONNX。不过目前tensorrt5.1.5版本不支持opset10，我会继续跟踪。

sudo apt-get install python3-pip protobuf-compiler libprotoc-dev
pip3 install onnx -i https://pypi.tuna.tsinghua.edu.cn/simple
pip3 install onnxruntime-gpu -i https://pypi.tuna.tsinghua.edu.cn/simple

onnx-tensorrt
首先升级cmake到3.13

sudo apt-get install libprotobuf-dev protobuf-compiler
git clone --recurse-submodules https://github.com/onnx/onnx-tensorrt.git
mkdir build
cd build 
cmake .. -DTENSORRT_ROOT=/home/xs/TensorRT-7.0.0.11
make -j8
sudo make install

keras-onnx

tensrrt5

由于过去项目不兼容，我重新使用了tensrrt5

//官网下载对于版本
tar xzvf TensorRT-5.1.5.0.Ubuntu-18.04.2.x86_64-gnu.cuda-10.0.cudnn7.5.tar.gz
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/xs/TensorRT-5.1.5.0/lib
cd TensorRT-5.1.5.0/python
sudo pip3 install tensorrt-5.1.5.0-cp36-none-linux_x86_64.whl
cd ../uff
sudo pip3 install uff-0.6.5-py2.py3-none-any.whl -i https://pypi.tuna.tsinghua.edu.cn/simple
cd ../graphsurgeon
sudo pip3 install graphsurgeon-0.4.1-py2.py3-none-any.whl
pip3 install tensorflow-gpu  -i https://pypi.tuna.tsinghua.edu.cn/simple 
cd ../lib
sudo cp libnvinfer.so.5 /usr/lib/
sudo cp libnvonnxparser.so.0 /usr/lib/
sudo cp libnvonnxparser_runtime.so.0 /usr/lib/
sudo cp libnvparsers.so.5 /usr/lib/
sudo cp libnvinfer_plugin.so.5 /usr/lib/
#onnx-tensorrt
git clone  -b v5.0 --recurse-submodules https://github.com/onnx/onnx-tensorrt.git

tensorrt-inference-server

这是tensorrt的服务器部署方案

docker pull nvcr.io/nvidia/tensorrtserver:19.08-py3

deepstream

这是tensorrt的嵌入式部署方案