bazel+tensorflow【未解决】+ubuntu20.04 源码安装

weixin_54470372

已于 2022-11-03 10:29:16 修改

阅读量1.4k

点赞数 1

分类专栏： install 文章标签： linux ubuntu

于 2022-10-27 12:01:32 首次发布

本文链接：https://blog.csdn.net/weixin_54470372/article/details/127534345

版权

install 专栏收录该内容

18 篇文章 5 订阅

订阅专栏

1、查询版本

2、安装bazel

（1）Install required packages

（1）从github上下载tensorflow

（2）配置

（3）构建pip软件包

【未解决】

1、查询版本

链接：从源代码构建 | TensorFlow

从上方链接查询bazel、tensorflow 、cuda的版本对应要求

2、安装bazel

选择使用二进制安装文件

链接：Installing Bazel on Ubuntu - Bazel 4.0.0

（1）Install required packages

sudo apt install g++ unzip zip
sudo apt-get install openjdk-11-jdk

（2）下载安装包

链接：https://github.com/bazelbuild/bazel/releases

其中，3.7.2的链接：https://github.com/bazelbuild/bazel/releases?q=3.7.2&expanded=true

在assets中找到

bazel-3.7.2-installer-linux-x86_64.sh

（3）安装

chmod +x bazel-<version>-installer-linux-x86_64.sh
./bazel-<version>-installer-linux-x86_64.sh --user

（4）配置路径

在~/.bashrc中加入

export PATH="$PATH:$HOME/bin"

（5）查询版本

$ bazel version
Starting local Bazel server and connecting to it...
Build label: 3.7.2
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Dec 17 16:57:23 2020 (1608224243)
Build timestamp: 1608224243
Build timestamp as int: 1608224243

3、安装tensorflow

官方文档：https://www.tensorflow.org/install/source

（1）从github上下载tensorflow

（2）配置

./configure

配置日志如下：

~/tools/tensorflow-r2.6$ ./configure
You have bazel 3.7.2 installed.
Please specify the location of python. [Default is /home/rakuli/anaconda3/bin/python3]: 


Found possible Python library paths:
  /home/rakuli/anaconda3/lib/python3.9/site-packages
Please input the desired Python library path to use.  Default is [/home/rakuli/anaconda3/lib/python3.9/site-packages]

Do you wish to build TensorFlow with ROCm support? [y/N]: n
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with TensorRT support? [y/N]: y
TensorRT support will be enabled for TensorFlow.

Could not find any NvInferVersion.h matching version '' in any subdirectory:
        ''
        'include'
        'include/cuda'
        'include/*-linux-gnu'
        'extras/CUPTI/include'
        'include/cuda/CUPTI'
        'local/cuda/extras/CUPTI/include'
of:
        '/lib'
        '/lib/x86_64-linux-gnu'
        '/lib32'
        '/usr'
        '/usr/lib/x86_64-linux-gnu/libfakeroot'
        '/usr/local/cuda'
        '/usr/local/cuda-11.2/targets/x86_64-linux/lib'
        '/usr/local/lib'

Asking for detailed CUDA configuration...

Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 10]: 11


Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7]: 8


Please specify the TensorRT version you want to use. [Leave empty to default to TensorRT 6]: 8


Please specify the locally installed NCCL version you want to use. [Leave empty to use http://github.com/nvidia/nccl]: 2


Please specify the comma-separated list of base paths to look for CUDA libraries and headers. [Leave empty to use the default]: /usr/lib/x86_64-linux-gnu,/usr/include,/usr/local/cuda-11.2,/usr/local/cuda-11.2/bin,/usr/local/cuda-11.2/lib64,/usr/local/cuda-11.2/include,/opt/TensorRT-8.2.1.8/bin,/opt/TensorRT-8.2.1.8/include,/opt/TensorRT-8.2.1.8/lib,/opt/TensorRT-8.2.1.8


Found CUDA 11.2 in:
    /usr/local/cuda-11.2/targets/x86_64-linux/lib
    /usr/local/cuda-11.2/targets/x86_64-linux/include
Found cuDNN 8 in:
    /usr/lib/x86_64-linux-gnu
    /usr/include
Found TensorRT 8 in:
    /opt/TensorRT-8.2.1.8/targets/x86_64-linux-gnu/lib
    /opt/TensorRT-8.2.1.8/include
Found NCCL 2 in:
    /usr/lib/x86_64-linux-gnu
    /usr/include


Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Each capability can be specified as "x.y" or "compute_xy" to include both virtual and binary GPU code, or as "sm_xy" to only include the binary code.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 7.5]: 


Do you want to use clang as CUDA compiler? [y/N]: n
nvcc will be used as CUDA compiler.

Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]: 


Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]: 


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: n
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
	--config=mkl         	# Build with MKL support.
	--config=mkl_aarch64 	# Build with oneDNN and Compute Library for the Arm Architecture (ACL).
	--config=monolithic  	# Config for mostly static monolithic build.
	--config=numa        	# Build with NUMA support.
	--config=dynamic_kernels	# (Experimental) Build kernels into separate shared objects.
	--config=v1          	# Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
	--config=nogcp       	# Disable GCP support.
	--config=nonccl      	# Disable NVIDIA NCCL support.
Configuration finished

存几个比较麻烦的地方：

需要CUDA
需要cuDNN：【最新】cuDNN在CUDA11.7+Ubuntu20.04下的安装及卸载_weixin_54470372的博客-CSDN博客
需要TensorRT：TensorRT 8+Ubuntu20.04安装_weixin_54470372的博客-CSDN博客
需要NCCL：NCCL+Ubuntu20.04安装_weixin_54470372的博客-CSDN博客
Please specify the comma-separated list of base paths to look for CUDA libraries and headers.

这个地方找了很久，可以通过whereis cudnn来找，还有一些我是通过查看以前的配置日志来找齐路径的，我的路径如下：

/usr/lib/x86_64-linux-gnu,/usr/include,/usr/local/cuda-11.2,/usr/local/cuda-11.2/bin,/usr/local/cuda-11.2/lib64,/usr/local/cuda-11.2/include,/opt/TensorRT-8.2.1.8/bin,/opt/TensorRT-8.2.1.8/include,/opt/TensorRT-8.2.1.8/lib,/opt/TensorRT-8.2.1.8

cudnn的路径是我找了最久的地方，可能因为我是deb文件安装的，需要的文件不在/usr/local/cuda-11.2，在这里放一下我的cudnn路径：

/usr/lib/x86_64-linux-gnu：存放libcudnn.so.8*

/usr/include：存放libcudnn.so.8*

（3）构建pip软件包

【未解决】

官方给出的命令：

bazel build --jobs 2 --config=opt //tensorflow/tools/pip_package:build_pip_package

但是报错：

ERROR: /home/r*/tools/tensorflow-r2.6/tensorflow/compiler/tf2tensorrt/BUILD:43:11: C++ compilation of rule '//tensorflow/compiler/tf2tensorrt:tensorrt_stub' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/tensorflow/compiler/tf2tensorrt/_objs/tensorrt_stub/nvinfer_stub.d ... (remaining 150 argument(s) skipped)

即使是增加了控制ram使用量的option也是同样的报错

bazel build --jobs 2 --config=opt //tensorflow/tools/pip_package:build_pip_package --local_ram_resources=1024

但是查询内存使用量发现可用的swap还有约1.3G，那么1024mb=1G是足够了的，暂时还没解决

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           15Gi       4.4Gi       3.1Gi       375Mi       8.0Gi        10Gi
Swap:         2.0Gi       679Mi       1.3Gi

详细日志如下

$ bazel build --jobs 2 --config=opt //tensorflow/tools/pip_package:build_pip_package
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=80
INFO: Reading rc options for 'build' from /home/rakuli/tools/tensorflow-r2.6/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/rakuli/tools/tensorflow-r2.6/.bazelrc:
  'build' options: --define framework_shared_object=true --java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --host_java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true
INFO: Reading rc options for 'build' from /home/rakuli/tools/tensorflow-r2.6/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/home/rakuli/anaconda3/bin/python3 --action_env PYTHON_LIB_PATH=/home/rakuli/anaconda3/lib/python3.9/site-packages --python_path=/home/rakuli/anaconda3/bin/python3 --config=tensorrt --action_env TF_CUDA_VERSION=11 --action_env TF_CUDNN_VERSION=8 --action_env TF_TENSORRT_VERSION=8 --action_env TF_NCCL_VERSION=2 --action_env TF_CUDA_PATHS=/usr/lib/x86_64-linux-gnu,/usr/include,/usr/local/cuda-11.2,/usr/local/cuda-11.2/bin,/usr/local/cuda-11.2/lib64,/usr/local/cuda-11.2/include,/opt/TensorRT-8.2.1.8/bin,/opt/TensorRT-8.2.1.8/include,/opt/TensorRT-8.2.1.8/lib,/opt/TensorRT-8.2.1.8 --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-11.2 --action_env TF_CUDA_COMPUTE_CAPABILITIES=7.5 --action_env LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64:/usr/local/cuda-11.2/lib64 --action_env GCC_HOST_COMPILER_PATH=/usr/bin/x86_64-linux-gnu-gcc-9 --config=cuda
INFO: Found applicable config definition build:short_logs in file /home/rakuli/tools/tensorflow-r2.6/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/rakuli/tools/tensorflow-r2.6/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:tensorrt in file /home/rakuli/tools/tensorflow-r2.6/.bazelrc: --repo_env TF_NEED_TENSORRT=1
INFO: Found applicable config definition build:cuda in file /home/rakuli/tools/tensorflow-r2.6/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:opt in file /home/rakuli/tools/tensorflow-r2.6/.tf_configure.bazelrc: --copt=-Wno-sign-compare --host_copt=-Wno-sign-compare
INFO: Found applicable config definition build:linux in file /home/rakuli/tools/tensorflow-r2.6/.bazelrc: --copt=-w --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --config=dynamic_kernels --distinct_host_configuration=false
INFO: Found applicable config definition build:dynamic_kernels in file /home/rakuli/tools/tensorflow-r2.6/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
DEBUG: /home/rakuli/.cache/bazel/_bazel_rakuli/db34c841133ec290f417a9b03fc74c5e/external/tf_runtime/third_party/cuda/dependencies.bzl:51:10: The following command will download NVIDIA proprietary software. By using the software you agree to comply with the terms of the license agreement that accompanies the software. If you do not agree to the terms of the license agreement, do not use the software.
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (9 packages loaded, 8392 targets configured).
INFO: Found 1 target...
ERROR: /home/rakuli/tools/tensorflow-r2.6/tensorflow/compiler/tf2tensorrt/BUILD:43:11: C++ compilation of rule '//tensorflow/compiler/tf2tensorrt:tensorrt_stub' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/tensorflow/compiler/tf2tensorrt/_objs/tensorrt_stub/nvinfer_stub.d ... (remaining 150 argument(s) skipped)
In file included from tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:58:
./tensorflow/compiler/tf2tensorrt/stub/NvInfer_5_1.inc:5:7: error: declaration of ‘void* createInferBuilder_INTERNAL(void*, int)’ has a different exception specifier
    5 | void* createInferBuilder_INTERNAL(void* logger, int version) {
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:17:
bazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers/third_party/tensorrt/NvInfer.h:8791:30: note: from previous declaration ‘void* createInferBuilder_INTERNAL(void*, int32_t) noexcept’
 8791 | extern "C" TENSORRTAPI void* createInferBuilder_INTERNAL(void* logger, int32_t version) noexcept;
      |                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:58:
./tensorflow/compiler/tf2tensorrt/stub/NvInfer_5_1.inc:12:7: error: declaration of ‘void* createInferRefitter_INTERNAL(void*, void*, int)’ has a different exception specifier
   12 | void* createInferRefitter_INTERNAL(void* engine, void* logger, int version) {
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from bazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers/third_party/tensorrt/NvInfer.h:54,
                 from tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:17:
bazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers/third_party/tensorrt/NvInferRuntime.h:2581:30: note: from previous declaration ‘void* createInferRefitter_INTERNAL(void*, void*, int32_t) noexcept’
 2581 | extern "C" TENSORRTAPI void* createInferRefitter_INTERNAL(void* engine, void* logger, int32_t version) noexcept;
      |                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:58:
./tensorflow/compiler/tf2tensorrt/stub/NvInfer_5_1.inc:19:7: error: declaration of ‘void* createInferRuntime_INTERNAL(void*, int)’ has a different exception specifier
   19 | void* createInferRuntime_INTERNAL(void* logger, int version) {
      |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from bazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers/third_party/tensorrt/NvInfer.h:54,
                 from tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:17:
bazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers/third_party/tensorrt/NvInferRuntime.h:2575:30: note: from previous declaration ‘void* createInferRuntime_INTERNAL(void*, int32_t) noexcept’
 2575 | extern "C" TENSORRTAPI void* createInferRuntime_INTERNAL(void* logger, int32_t version) noexcept;
      |                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:58:
./tensorflow/compiler/tf2tensorrt/stub/NvInfer_5_1.inc:26:20: error: declaration of ‘nvinfer1::ILogger* getLogger()’ has a different exception specifier
   26 | nvinfer1::ILogger* getLogger() {
      |                    ^~~~~~~~~
In file included from bazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers/third_party/tensorrt/NvInfer.h:54,
                 from tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:17:
bazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers/third_party/tensorrt/NvInferRuntime.h:2593:43: note: from previous declaration ‘nvinfer1::ILogger* getLogger() noexcept’
 2593 | extern "C" TENSORRTAPI nvinfer1::ILogger* getLogger() noexcept;
      |                                           ^~~~~~~~~
In file included from tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:58:
./tensorflow/compiler/tf2tensorrt/stub/NvInfer_5_1.inc:33:5: error: declaration of ‘int getInferLibVersion()’ has a different exception specifier
   33 | int getInferLibVersion() {
      |     ^~~~~~~~~~~~~~~~~~
In file included from bazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers/third_party/tensorrt/NvInferLegacyDims.h:53,
                 from bazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers/third_party/tensorrt/NvInfer.h:53,
                 from tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:17:
bazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers/third_party/tensorrt/NvInferRuntimeCommon.h:1883:32: note: from previous declaration ‘int32_t getInferLibVersion() noexcept’
 1883 | extern "C" TENSORRTAPI int32_t getInferLibVersion() noexcept;
      |                                ^~~~~~~~~~~~~~~~~~
In file included from tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:58:
./tensorflow/compiler/tf2tensorrt/stub/NvInfer_5_1.inc:40:28: error: declaration of ‘nvinfer1::IPluginRegistry* getPluginRegistry()’ has a different exception specifier
   40 | nvinfer1::IPluginRegistry* getPluginRegistry() {
      |                            ^~~~~~~~~~~~~~~~~
In file included from bazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers/third_party/tensorrt/NvInfer.h:54,
                 from tensorflow/compiler/tf2tensorrt/stub/nvinfer_stub.cc:17:
bazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers/third_party/tensorrt/NvInferRuntime.h:2586:51: note: from previous declaration ‘nvinfer1::IPluginRegistry* getPluginRegistry() noexcept’
 2586 | extern "C" TENSORRTAPI nvinfer1::IPluginRegistry* getPluginRegistry() noexcept;
      |                                                   ^~~~~~~~~~~~~~~~~
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1885.413s, Critical Path: 38.04s
INFO: 3596 processes: 48 internal, 3548 local.
FAILED: Build did NOT complete successfully

参考：

（1）bazel安装

在 Ubuntu 上安装 Bazel_喵先生!的博客-CSDN博客_ubuntu bazel更细

（2）TensorRT安装

TF2.x+TensorRT 环境配置_ssmixi的博客-CSDN博客_could not find any nvinferversion.h matching versi

Tensorflow: TensoeRT的安装及编译（c++接口）_Felaim的博客-CSDN博客

（3）tensorflow安装

TensorFlow1.15源码安装 - 知乎

weixin_54470372

关注

1
点赞
踩
5

收藏

觉得还不错? 一键收藏
0
评论
bazel+tensorflow【未解决】+ubuntu20.04 源码安装

bazel+tensorflow+ubuntu20.04 源码安装crosstool_wrapper_driver_is_not_gcc failed: error executing commandexternal/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF
复制链接

扫一扫