本文将介绍如何编译tensorflow+cuda+cudnn+tensorrt
环境查看
tensorflow版本要求:https://www.tensorflow.org/install/source
TennsorRT环境要求:
https://docs.nvidia.com/deeplearning/tensorrt/archives/index.html
选择
综合对比确定环境如下:
项目 | Value |
---|---|
cuda | 10.1 update2 |
cudnn | 7.6.5 |
bazel | 3.1.0 |
tensorrt | 6.0.1 |
tensorflow | 2.3.0 |
gcc g++ | 7.3 |
numpy | <1.19 |
一:安装cuda10.1 update2
注意:需要安装最新NVIDIA驱动
下载地址:https://developer.nvidia.com/cuda-toolkit-archive
执行代码:
wget https://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.243_418.87.00_linux.run
sudo sh cuda_10.1.243_418.87.00_linux.run
配置CUDA相关环境变量
修改/.bashrc文件(或/.zhsrc) 在文件末尾添加环境变量
Tensorflow官方安装历程要求注意的是:配置PATH和LD_LIBRARY_PATH和CUDA_HOME环境变量.
vim ~/.bashrc #修改配置文件(如果你用的是zsh,则需要修改 ~/.zshrc文件)
#在文件结尾处添加
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
export PATH=$PATH:/usr/local/cuda/bin
export CUDA_HOME=/usr/local/cuda
补充:
- 若文件下载较慢,可通过以下命令设置终端代理
export http_proxy=http://127.0.0.1:12333
export https_proxy=http://127.0.0.1:12333
- 若你的系统中存在多个cuda版本,可通过以下教程进行cuda版本切换
https://blog.csdn.net/yinxingtianxia/article/details/80462892
https://blog.csdn.net/wuprogrammer/article/details/108942268
二:cudnn安装
下载地址:https://developer.nvidia.com/rdp/cudnn-archive
然后解压,并进入到相应目录,运行以下命令:
sudo cp cuda/include/cudnn.h /usr/local/cuda/include
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h
sudo chmod a+r /usr/local/cuda/lib64/libcudnn*
PS:cudnn版本查询:
cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
PS2:cudnn8 中的部分安装步骤有所改动,请读者参考最新的安装教程https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html
三:GCC版本切换
执行以下命令查看gcc版本
gcc --version
若gcc版本不符合要求:
按照以下顺序下载相应版本:
- 使用以下命令将 ubuntu-toolchain-r/test PPA添加到 Ubuntu 系统:
sudo apt install software-properties-common
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
- 执行以下命令安装所需的 GCC 和 G ++ 版本:
(如果无法定位安装包,可以在设置里换个安装源)
sudo apt install gcc-7 g++-7
以下命令将为每个版本配置替代方案并将优先级与其关联,默认版本是具有最高优先级的版本,在我们的例子中是 gcc-7:
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 90 --slave /usr/bin/g++ g++ /usr/bin/g++-7
- 如果要更改默认版本,可以使用 update-alternatives 命令:
sudo update-alternatives --config gcc
四:安装tensorRT
下载地址:
https://developer.nvidia.com/zh-cn/tensorrt
解压文件夹
在.bashrc文件里添加绝对路径
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<eg:TensorRT-6.x.x.x/lib>
终端进入相应文件夹(若由anaconda虚拟环境,则在虚拟环境下执行以下安装命令):
1.安装TensorRT wheel文件
cd TensorRT-6.x.x.x/python
sudo pip3 install tensorrt-6.x.x.x-cp3x-none-linux_x86_64.whl
2.安装uff
cd TensorRT-6.x.x.x/uff
sudo pip3 install uff-0.6.5-py2.py3-none-any.whl
3.安装graphsurgeon :
cd TensorRT-6.x.x.x/graphsurgeon
sudo pip3 install graphsurgeon-0.4.1-py2.py3-none-any.whl
五:安装bazel
下载地址:https://github.com/bazelbuild/bazel/releases
chmod +x bazel-<version>-installer-linux-x86_64.sh
./bazel-<version>-installer-linux-x86_64.sh --user
在.bashrc文件夹中添加环境变量
export PATH="$PATH:$HOME/bin"
六:安装tensorflow
若有虚拟环境,进入虚拟环境后执行以下命令
安装 预处理包
pip install -U --user pip wheel
pip install numpy==1.16
pip install -U --user keras_preprocessing --no-deps
拉取tensorflow源码
git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
git checkout r2.3
./configure
接下来,配置系统会给出各种询问,以确认编译时的配置参数:
参数具体解释:https://blog.csdn.net/yhily2008/article/details/79967118
spicker@spicker-1:~/software/tensorflow$ ./configure
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.25.2 installed.
Please specify the location of python. [Default is /home/spicker/anaconda3/bin/python]:
Found possible Python library paths:
/home/spicker/anaconda3/lib/python3.6/site-packages
Please input the desired Python library path to use. Default is [/home/spicker/anaconda3/lib/python3.6/site-packages]
Do you wish to build TensorFlow with XLA JIT support? [Y/n]: n
No XLA JIT support will be enabled for TensorFlow.
Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: N
No OpenCL SYCL support will be enabled for TensorFlow.
Do you wish to build TensorFlow with ROCm support? [y/N]: N
No ROCm support will be enabled for TensorFlow.
Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.
Do you wish to build TensorFlow with TensorRT support? [y/N]: y
TensorRT support will be enabled for TensorFlow.
Found CUDA 10.1 in:
/usr/local/cuda/lib64
/usr/local/cuda/include
Found cuDNN 7.6.3 in:
/usr/local/cuda/lib64
/usr/local/cuda/include
Found TensorRT 5 in:
/usr/lib
/usr/include
Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 6.1]:
Do you want to use clang as CUDA compiler? [y/N]: N
nvcc will be used as CUDA compiler.
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Do you wish to build TensorFlow with MPI support? [y/N]: N
No MPI support will be enabled for TensorFlow.
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native -Wno-sign-compare]:
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: N
Not configuring the WORKSPACE for Android builds.
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
--config=mkl # Build with MKL support.
--config=monolithic # Config for mostly static monolithic build.
--config=gdr # Build with GDR support.
--config=verbs # Build with libverbs support.
--config=ngraph # Build with Intel nGraph support.
--config=numa # Build with NUMA support.
--config=dynamic_kernels # (Experimental) Build kernels into separate shared objects.
Preconfigured Bazel build configs to DISABLE default on features:
--config=noaws # Disable AWS S3 filesystem support.
--config=nogcp # Disable GCP support.
--config=nohdfs # Disable HDFS support.
--config=noignite # Disable Apache Ignite support.
--config=nokafka # Disable Apache Kafka support.
--config=nonccl # Disable NVIDIA NCCL support.
Configuration finished
补充:若config文件找不到cuda,cudnn,tensorRT文件如下图:
felaim@felaim-pc:~/Documents/software/tensorflow-r1.14$ ./configure
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.24.1 installed.
Please specify the location of python. [Default is /usr/bin/python]: /home/felaim/anaconda3/envs/tensorflow-gpu/bin/python
Found possible Python library paths:
/home/felaim/anaconda3/envs/tensorflow-gpu/lib/python3.7/site-packages
Please input the desired Python library path to use. Default is [/home/felaim/anaconda3/envs/tensorflow-gpu/lib/python3.7/site-packages]
Do you wish to build TensorFlow with XLA JIT support? [Y/n]: n
No XLA JIT support will be enabled for TensorFlow.
Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: n
No OpenCL SYCL support will be enabled for TensorFlow.
Do you wish to build TensorFlow with ROCm support? [y/N]: n
No ROCm support will be enabled for TensorFlow.
Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.
Do you wish to build TensorFlow with TensorRT support? [y/N]: y
TensorRT support will be enabled for TensorFlow.
Could not find any NvInferVersion.h matching version '' in any subdirectory:
''
'include'
'include/cuda'
'include/*-linux-gnu'
'extras/CUPTI/include'
'include/cuda/CUPTI'
of:
'/home/felaim/intel/ipp/lib/intel64_lin'
'/lib/x86_64-linux-gnu'
'/usr'
'/usr/lib'
'/usr/lib/x86_64-linux-gnu'
'/usr/lib/x86_64-linux-gnu/libfakeroot'
'/usr/lib/x86_64-linux-gnu/mesa'
'/usr/lib/x86_64-linux-gnu/mesa-egl'
'/usr/local/cuda'
'/usr/local/cuda-10.1/targets/x86_64-linux/lib'
'/usr/local/cuda/extras/CUPTI/lib64'
'/usr/local/lib'
Asking for detailed CUDA configuration...
Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 10]:
Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7]:
Please specify the TensorRT version you want to use. [Leave empty to default to TensorRT 5]: 6
Please specify the locally installed NCCL version you want to use. [Leave empty to use http://github.com/nvidia/nccl]:
Please specify the comma-separated list of base paths to look for CUDA libraries and headers. [Leave empty to use the default]: /usr/local/cuda,/usr/local/cuda/bin,/usr/local/cuda/lib64,/usr/local/cuda/include,/home/felaim/Documents/software/TensorRT-6.0.1.5,/usr/lib/x86_64-linux-gnu/
Found CUDA 10.1 in:
/usr/local/cuda/lib64
/usr/local/cuda/include
Found cuDNN 7 in:
/usr/local/cuda/lib64
/usr/local/cuda/include
Found TensorRT 6 in:
/home/felaim/Documents/software/TensorRT-6.0.1.5/lib
/home/felaim/Documents/software/TensorRT-6.0.1.5/include
Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 5.2]:
Do you want to use clang as CUDA compiler? [y/N]:
nvcc will be used as CUDA compiler.
Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]:
Do you wish to build TensorFlow with MPI support? [y/N]:
No MPI support will be enabled for TensorFlow.
Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native -Wno-sign-compare]:
Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]:
Not configuring the WORKSPACE for Android builds.
Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
--config=mkl # Build with MKL support.
--config=monolithic # Config for mostly static monolithic build.
--config=gdr # Build with GDR support.
--config=verbs # Build with libverbs support.
--config=ngraph # Build with Intel nGraph support.
--config=numa # Build with NUMA support.
--config=dynamic_kernels # (Experimental) Build kernels into separate shared objects.
Preconfigured Bazel build configs to DISABLE default on features:
--config=noaws # Disable AWS S3 filesystem support.
--config=nogcp # Disable GCP support.
--config=nohdfs # Disable HDFS support.
--config=noignite # Disable Apache Ignite support.
--config=nokafka # Disable Apache Kafka support.
--config=nonccl # Disable NVIDIA NCCL support.
Configuration finished
可在配置过程输入文件位置解决,如:
/usr/local/cuda,/usr/local/cuda/bin,/usr/local/cuda/lib64,/usr/local/cuda/include,/usr/lib/x86_64-linux-gnu/,/home/hove/TensorRT-6.0.1.5
编译源码:
若配置较好
bazel build --config=cuda --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --verbose_failures //tensorflow/tools/pip_package:build_pip_package
较差
bazel build --config=cuda --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --local_ram_resources=4096 --local_cpu_resources=3 --verbose_failures //tensorflow/tools/pip_package:build_pip_package
- –cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" 是让gcc5.0以后的版本对源码兼容
- –local_ram_resources=4096 --local_cpu_resources=3分别限制编译过程占用的内存和核数
- –verbose_failures 将编译过程产生的错误显示出来
在tensorflow文件夹中运行
./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
备注:若每次编译失败需要清除bazel缓存,重新编译
rm -rf ~/.cache/bazel/*
安装pip文件:
pip install /tmp/tensorflow_pkg/tensorflow-version-tags.whl
大功告成!