linux离线安装stb,离线源码安装tensorflow

最新推荐文章于 2022-06-13 13:01:03 发布

我要WhatYouNeed

最新推荐文章于 2022-06-13 13:01:03 发布

阅读量2.1k

点赞数

文章标签： linux离线安装stb

离线源码安装tensorflow

离线源码编译简直是个坑，记录下我踩坑的过程。。

参考：

http://www.icode9.com/content-1-97054.html

https://blog.csdn.net/conling_/article/details/72902571

https://github.com/tensorflow/tensorflow/issues/15889

环境：

CentOS Linux release 7.2

CUDA 9.2

CUDNN 7.4.2

NCCL 2.3.7

1. 在本机安装一个http的服务

// 查询安装：

rpm -q httpd1

// 安装apache httpd

yum install httpd

2. 将所有要下载的文件放到DocumentRoot中，默认的目录是/var/www/html

所有要下载的文件见

tensorflow/WORKSPACE

tensorflow/workspace.bzl

基本是以下这一些(TensorFlow 1.9)

https://mirror.bazel.build/github.com/intel/mkl-dnn/releases/download/v0.14/mklml_win_2018.0.3.20180406.zip

https://mirror.bazel.build/github.com/intel/mkl-dnn/releases/download/v0.14/mklml_mac_2018.0.3.20180406.tgz

https://mirror.bazel.build/github.com/intel/mkl-dnn/archive/v0.14.tar.gz

https://mirror.bazel.build/github.com/abseil/abseil-cpp/archive/9613678332c976568272c8f4a78631a29159271d.tar.gz

https://mirror.bazel.build/bitbucket.org/eigen/eigen/get/fd6845384b86.tar.gz

https://mirror.bazel.build/github.com/raspberrypi/tools/archive/0e906ebc527eab1cdbf7adabff5b474da9562e9f.tar.gz

https://mirror.bazel.build/github.com/hfp/libxsmm/archive/1.8.1.tar.gz

https://mirror.bazel.build/github.com/google/or-tools/archive/253f7955c6a1fd805408fba2e42ac6d45b312d15.tar.gz

https://mirror.bazel.build/github.com/google/re2/archive/26cd968b735e227361c9703683266f01e5df7857.tar.gz

https://mirror.bazel.build/github.com/google/gemmlowp/archive/38ebac7b059e84692f53e5938f97a9943c120d98.zip

https://mirror.bazel.build/github.com/google/farmhash/archive/816a4ae622e964763ca0862d9dbd19324a1eaf45.tar.gz

http://mirror.bazel.build/github.com/google/highwayhash/archive/fd3d9af80465e4383162e4a7c5e2f406e82dd968.tar.gz

https://mirror.bazel.build/www.nasm.us/pub/nasm/releasebuilds/2.12.02/nasm-2.12.02.tar.bz2

https://mirror.bazel.build/github.com/libjpeg-turbo/libjpeg-turbo/archive/1.5.3.tar.gz

https://mirror.bazel.build/github.com/glennrp/libpng/archive/v1.6.34.tar.gz

https://mirror.bazel.build/www.sqlite.org/2018/sqlite-amalgamation-3230100.zip

https://mirror.bazel.build/ufpr.dl.sourceforge.net/project/giflib/giflib-5.1.4.tar.gz

https://mirror.bazel.build/pypi.python.org/packages/source/s/six/six-1.10.0.tar.gz

https://mirror.bazel.build/pypi.python.org/packages/d8/be/c4276b3199ec3feee2a88bc64810fbea8f26d961e0a4cd9c68387a9f35de/astor-0.6.2.tar.gz

https://mirror.bazel.build/pypi.python.org/packages/5c/78/ff794fcae2ce8aa6323e789d1f8b3b7765f601e7702726f430e814822b96/gast-0.2.0.tar.gz

https://mirror.bazel.build/pypi.python.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz

https://mirror.bazel.build/github.com/abseil/abseil-py/archive/ea8c4d2ddbf3fba610c4d613260561699b776db8.tar.gz

https://mirror.bazel.build/pypi.python.org/packages/bc/cc/3cdb0a02e7e96f6c70bd971bc8a90b8463fda83e264fa9c5c1c98ceabd81/backports.weakref-1.0rc1.tar.gz

https://mirror.bazel.build/docs.python.org/2.7/_sources/license.txt

https://mirror.bazel.build/github.com/google/protobuf/archive/396336eb961b75f03b25824fe86cf6490fb75e3a.tar.gz

https://mirror.bazel.build/github.com/google/nsync/archive/0559ce013feac8db639ee1bf776aca0325d28777.tar.gz

https://mirror.bazel.build/github.com/google/googletest/archive/9816b96a6ddc0430671693df90192bbee57108b6.zip

https://mirror.bazel.build/github.com/gflags/gflags/archive/f8a0efe03aa69b3336d8e228b37d4ccb17324b88.tar.gz

https://mirror.bazel.build/ftp.exim.org/pub/pcre/pcre-8.39.tar.gz

https://mirror.bazel.build/ufpr.dl.sourceforge.net/project/swig/swig/swig-3.0.8/swig-3.0.8.tar.gz

https://mirror.bazel.build/curl.haxx.se/download/curl-7.49.1.tar.gz

https://mirror.bazel.build/github.com/grpc/grpc/archive/d184fa229d75d336aedea0041bd59cb93e7e267f.tar.gz

https://mirror.bazel.build/github.com/antirez/linenoise/archive/c894b9e59f02203dbe4e2be657572cf88c4230c3.tar.gz

https://mirror.bazel.build/github.com/llvm-mirror/llvm/archive/bf13d093f13a295d71080614c3036ada591201d5.tar.gz

https://mirror.bazel.build/github.com/LMDB/lmdb/archive/LMDB_0.9.19.tar.gz

https://mirror.bazel.build/github.com/open-source-parsers/jsoncpp/archive/11086dd6a7eba04289944367ca82cea71299ed70.tar.gz

https://mirror.bazel.build/github.com/google/boringssl/archive/a0fb951d2a26a8ee746b52f3ba81ab011a0af778.tar.gz

https://mirror.bazel.build/zlib.net/zlib-1.2.11.tar.gz

https://mirror.bazel.build/www.kurims.kyoto-u.ac.jp/~ooura/fft.tgz

https://mirror.bazel.build/github.com/google/snappy/archive/1.1.7.tar.gz

https://mirror.bazel.build/github.com/nvidia/nccl/archive/03d856977ecbaac87e598c0c4bafca96761b9ac7.tar.gz

https://mirror.bazel.build/github.com/edenhill/librdkafka/archive/v0.11.1.tar.gz

https://mirror.bazel.build/github.com/aws/aws-sdk-cpp/archive/1.3.15.tar.gz

https://mirror.bazel.build/repo1.maven.org/maven2/junit/junit/4.12/junit-4.12.jar

https://mirror.bazel.build/repo1.maven.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar

https://mirror.bazel.build/github.com/jemalloc/jemalloc/archive/4.4.0.tar.gz

http://mirror.bazel.build/repo1.maven.org/maven2/com/google/testing/compile/compile-testing/0.11/compile-testing-0.11.jar

http://mirror.bazel.build/repo1.maven.org/maven2/com/google/truth/truth/0.32/truth-0.32.jar

http://mirror.bazel.build/repo1.maven.org/maven2/org/checkerframework/checker-qual/2.4.0/checker-qual-2.4.0.jar

https://mirror.bazel.build/github.com/google/pprof/archive/c0fb62ec88c411cc91194465e54db2632845b650.tar.gz

https://mirror.bazel.build/github.com/NVlabs/cub/archive/1.8.0.zip

https://mirror.bazel.build/github.com/cython/cython/archive/3732784c45cfb040a5b0936951d196f83a12ea17.tar.gz

https://mirror.bazel.build/github.com/bazelbuild/bazel-toolchains/archive/44200e0c026d86c53470d107b3697a3e46469c43.tar.gz

https://mirror.bazel.build/github.com/intel/ARM_NEON_2_x86_SSE/archive/0f77d9d182265259b135dad949230ecbf1a2633d.tar.gz

https://mirror.bazel.build/github.com/google/flatbuffers/archive/971a68110e4fc1bace10fcb6deeb189e7e1a34ce.tar.gz

https://mirror.bazel.build/storage.googleapis.com/download.tensorflow.org/models/tflite/mobilenet_v1_224_android_quant_2017_11_08.zip

https://mirror.bazel.build/storage.googleapis.com/download.tensorflow.org/models/tflite/mobilenet_ssd_tflite_v1.zip

https://mirror.bazel.build/storage.googleapis.com/download.tensorflow.org/models/tflite/conv_actions_tflite.zip

https://mirror.bazel.build/storage.googleapis.com/download.tensorflow.org/models/tflite/smartreply_1.0_2017_11_01.zip

https://mirror.bazel.build/storage.googleapis.com/download.tensorflow.org/data/ovic.zip

全部下载下来 down.sh

#!/bin/bash

for i in `cat packages.txt`

wget $i

echo "$i"

done

3. 修改tensorflow/WORKSPACE、tensorflow/workspace.bzl文件

在urls中加入http://localhost/xxx这一项，例如：

tf_http_archive(

name = "com_google_absl",

urls = [

"https://mirror.bazel.build/github.com/abseil/abseil-cpp/archive/9613678332c976568272c8f4a78631a29159271d.tar.gz",

"https://github.com/abseil/abseil-cpp/archive/9613678332c976568272c8f4a78631a29159271d.tar.gz",

"http://localhost/9613678332c976568272c8f4a78631a29159271d.tar.gz",

sha256 = "1273a1434ced93bc3e703a48c5dced058c95e995c8c009e9bdcb24a69e2180e9",

strip_prefix = "abseil-cpp-9613678332c976568272c8f4a78631a29159271d",

build_file = clean_dep("//third_party:com_google_absl.BUILD"),

)

tf_http_archive(

name = "eigen_archive",

urls = [

"https://mirror.bazel.build/bitbucket.org/eigen/eigen/get/fd6845384b86.tar.gz",

"https://bitbucket.org/eigen/eigen/get/fd6845384b86.tar.gz",

"http://localhost/fd6845384b86.tar.gz",

sha256 = "d956415d784fa4e42b6a2a45c32556d6aec9d0a3d8ef48baee2522ab762556a9",

strip_prefix = "eigen-eigen-fd6845384b86",

build_file = clean_dep("//third_party:eigen.BUILD"),

)

4. 编译及生成whl包

bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

报错1：

Starting local Bazel server and connecting to it...

..........

WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.

ERROR: /home/tf-install/tensorflow/tensorflow/tools/pip_package/BUILD:168:1: error loading package 'tensorflow': Encountered error while reading extension file 'protobuf.bzl': no such package '@protobuf_archive//': java.io.IOException: Error downloading [https://mirror.bazel.build/github.com/google/protobuf/archive/396336eb961b75f03b25824fe86cf6490fb75e3a.tar.gz, https://github.com/google/protobuf/archive/396336eb961b75f03b25824fe86cf6490fb75e3a.tar.gz] to /root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/external/protobuf_archive/396336eb961b75f03b25824fe86cf6490fb75e3a.tar.gz: All mirrors are down: [Network is unreachable (connect failed)] and referenced by '//tensorflow/tools/pip_package:build_pip_package'

ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted: error loading package 'tensorflow': Encountered error while reading extension file 'protobuf.bzl': no such package '@protobuf_archive//': java.io.IOException: Error downloading [https://mirror.bazel.build/github.com/google/protobuf/archive/396336eb961b75f03b25824fe86cf6490fb75e3a.tar.gz, https://github.com/google/protobuf/archive/396336eb961b75f03b25824fe86cf6490fb75e3a.tar.gz] to /root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/external/protobuf_archive/396336eb961b75f03b25824fe86cf6490fb75e3a.tar.gz: All mirrors are down: [Network is unreachable (connect failed)]

INFO: Elapsed time: 30.697s

INFO: 0 processes.

在/var/www/html中已经有396336eb961b75f03b25824fe86cf6490fb75e3a.tar.gz了，但不知道为啥就是报错，尝试过把396336eb961b75f03b25824fe86cf6490fb75e3a.tar.gz拷贝到/root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/external/protobuf_archive/目录下，但编译时该目录就会自动清空。

最后的解决方法是将解压后的文件拷贝到/root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/external/protobuf_archive/目录下，就可以了

报错2：

ERROR: /home/tf-install/tensorflow/tensorflow/tools/pip_package/BUILD:118:1: no such package '@double_conversion//': Error downloading [https://github.com/google/double-conversion/archive/mobilenet_v1_224_android_quant_2017_11_08.zip, http://localhost/mobilenet_v1_224_android_quant_2017_11_08.zip] to /root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/external/double_conversion/mobilenet_v1_224_android_quant_2017_11_08.zip: All mirrors are down: [GET returned 404 Not Found, GET returned 503 Service Unavailable] and referenced by '//tensorflow/tools/pip_package:licenses'

ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted: no such package '@double_conversion//': Error downloading [https://github.com/google/double-conversion/archive/mobilenet_v1_224_android_quant_2017_11_08.zip, http://localhost/mobilenet_v1_224_android_quant_2017_11_08.zip] to /root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/external/double_conversion/mobilenet_v1_224_android_quant_2017_11_08.zip: All mirrors are down: [GET returned 404 Not Found, GET returned 503 Service Unavailable]

INFO: Elapsed time: 19.812s

INFO: 0 processes.

FAILED: Build did NOT complete successfully (169 packages loaded)

这个错误就是。。tensorflow/workspace.bzl 这个文件里double-conversion这一部分写错了。。所以一定要细心！！！

报错3：

ERROR: missing input file '@local_config_nccl//:nccl/NCCL-SLA.txt'

ERROR: /home/tf-install/tensorflow/tensorflow/contrib/lite/toco/python/BUILD:34:1: Creating runfiles tree bazel-out/k8-opt/bin/tensorflow/contrib/lite/toco/python/toco_from_protos.runfiles failed: Process terminated by signal 15: Process terminated by signal 15

ERROR: /home/tf-install/tensorflow/tensorflow/tools/pip_package/BUILD:168:1: //tensorflow/tools/pip_package:build_pip_package: missing input file '@local_config_nccl//:nccl/NCCL-SLA.txt'

Target //tensorflow/tools/pip_package:build_pip_package failed to build

Use --verbose_failures to see the command lines of failed build steps.

ERROR: /home/tf-install/tensorflow/tensorflow/tools/pip_package/BUILD:168:1 1 input file(s) do not exist

INFO: Elapsed time: 27.084s, Critical Path: 9.10s

INFO: 4 processes: 4 local.

FAILED: Build did NOT complete successfully

发现nccl2.3.7没有NCCL-SLA.txt文件(nccl2.3.5也没有)，又发现nccl2.1.12和nccl2.1.15的NCCL-SLA.txt文件是完全一致的，而且该文件就是一个Software License Agreement文件。因此就直接把一个文件拷贝到了nccl2.3.7目录处

报错4：

ERROR: /home/tf-install/tensorflow/tensorflow/contrib/stateless/BUILD:12:1: Linking of rule '//tensorflow/contrib/stateless:gen_stateless_random_ops_py_wrappers_cc' failed (Exit 1)

/usr/bin/ld: warning: libcublas.so.9.2, needed by bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sstateless_Cgen_Ustateless_Urandom_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)

/usr/bin/ld: warning: libcufft.so.9.2, needed by bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sstateless_Cgen_Ustateless_Urandom_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)

/usr/bin/ld: warning: libcurand.so.9.2, needed by bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sstateless_Cgen_Ustateless_Urandom_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)

bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sstateless_Cgen_Ustateless_Urandom_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cufftMakePlanMany@libcufft.so.9.2'

collect2: error: ld returned 1 exit status

Target //tensorflow/tools/pip_package:build_pip_package failed to build

Use --verbose_failures to see the command lines of failed build steps.

INFO: Elapsed time: 205.491s, Critical Path: 105.31s

INFO: 5449 processes: 5449 local.

FAILED: Build did NOT complete successfully

这里是因为没有CUDA的环境变量。。在~/.bashrc中加上

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64/:$LD_LIBRARY_PATH

export PATH=/usr/local/cuda/bin:$PATH

报错5：

Executing genrule //tensorflow/cc:no_op_genrule failed (Exit 127) libcublas.so.9.2: cannot open shared object file:

完整的错误没有保存，大概就是这里

解决方式：

echo "/usr/local/cuda/lib64" > /etc/ld.so.conf.d/cuda.conf

ldconfig

报错6：

cannot create symbolic link bazel-out -> /root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/execroot/org_tensorflow/bazel-out: /home/tf-install/tensorflow/bazel-out (File exists)

cannot create symbolic link bazel-tensorflow -> /root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/execroot/org_tensorflow: /home/tf-install/tensorflow/bazel-tensorflow (File exists)

cannot create symbolic link bazel-bin -> /root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/execroot/org_tensorflow/bazel-out/k8-opt/bin: /home/tf-install/tensorflow/bazel-bin (File exists)

cannot create symbolic link bazel-testlogs -> /root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/execroot/org_tensorflow/bazel-out/k8-opt/testlogs: /home/tf-install/tensorflow/bazel-testlogs (File exists)

cannot create symbolic link bazel-genfiles -> /root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/execroot/org_tensorflow/bazel-out/k8-opt/genfiles: /home/tf-install/tensorflow/bazel-genfiles (File exists)

Target //tensorflow/tools/pip_package:build_pip_package up-to-date:

/root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/pip_package/build_pip_package

虽然编译时会显示Build completed successfully，

但是生成whl包 bazel-bin/tensorflow/tools/pip_package/build_pip_package /home/tf-install时，会报错:

bazel-bin/tensorflow/tools/pip_package/build_pip_package 找不到这个文件，这个错误上边也显示了，没有链接成功。

bazel clean

bazel build

仍然是build成功，但是生成whl包时报错：

Target //tensorflow/tools/pip_package:build_pip_package up-to-date:

/root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/pip_package/build_pip_package

INFO: Elapsed time: 711.604s, Critical Path: 239.12s

INFO: 10805 processes: 10805 local.

INFO: Build completed successfully, 13734 total actions

/root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/execroot/org_tensorflow/bazel-out/k8-opt/bin/tensorflow/tools/pip_package/build_pip_package /data1/cathy/tf-install/

Tue Jan 8 11:37:48 CST 2019 : === Preparing sources in dir: /tmp/tmp.ov2ovDsqfm

Could not find bazel-bin. Did you run from the root of the build tree?

尝试自己建立软链接，但也报错：

ln -s /root/.cache/bazel/_bazel_root/de0cb75ac2885a31b458bd10d630147b/execroot/org_tensorflow/bazel-out bazel-out

ln: target 'bazel-out' is not a directory

最后，删掉tensorflow目录下的bazel-out、bazel-bin等目录，重新编译，终于成功啦！！！

####5. 安装

pip uninstall -y tensorflow

pip install tensorflow-1.9.0-cp27-cp27mu-linux_x86_64.whl

验证

python

Python 2.7.5 (default, Nov 20 2015, 02:00:19)