使用 virtualenv 源码安装 TensorFlow 1.4.1

由于TensorFlow1.5.0测试中多机nccl/xring时一直出错，故尝试之前版本，为了不破坏当前的环境，使用virtualenv来安装TensorFlow1.4.1，以下是安装过程

1. 安装所有必备工具：

apt-get install python-pip python-dev python-virtualenv

2. 建立一个全新的 virtualenv 环境. 将环境建在 /root/vir-tensorflow-1.4.1目录下, 执行:

virtualenv --system-site-packages /root/vir-tensorflow-1.4.1

此时出错：

HTTPError: 404 Client Error: Not Found for url: http://mirrors.aliyun.com/pypi/simple/pkg-resources/

修改python的镜像源 （参考 https://yq.aliyun.com/ask/56175/）

vim ~/.pip/pip.conf_virtualenv

[global]
index-url=http://blog.xiayf.cn/2013/11/01/change-pypi-mirror/

3. 激活 virtualenv

cd /root/vir-tensorflow-1.4.1
source bin/activate 

4. 源码安装TensorFlow，下载tensorflow-1.4.1后解压，我的环境是Ubuntu16.04 + CUDA9.1 + CUDNN7 + GCC5.4.0

4.1 配置文件

cd tensorflow-1.4.1
./configure

可以通过查看 .tf_configure.bazelrc 文件来看选择的配置，我的配置如下：

build --action_env PYTHON_BIN_PATH="/root/vir-tensorflow-1.4.1/bin/python"
build --action_env PYTHON_LIB_PATH="/root/vir-tensorflow-1.4.1/lib/python2.7/site-packages"
build --define PYTHON_BIN_PATH="/root/vir-tensorflow-1.4.1/bin/python"
build --define PYTHON_LIB_PATH="/root/vir-tensorflow-1.4.1/lib/python2.7/site-packages"
build --force_python=py2
build --host_force_python=py2
build --python_path="/root/vir-tensorflow-1.4.1/bin/python"
test --force_python=py2
test --host_force_python=py2
test --define PYTHON_BIN_PATH="/root/vir-tensorflow-1.4.1/bin/python"
test --define PYTHON_LIB_PATH="/root/vir-tensorflow-1.4.1/lib/python2.7/site-packages"
run --define PYTHON_BIN_PATH="/root/vir-tensorflow-1.4.1/bin/python"
run --define PYTHON_LIB_PATH="/root/vir-tensorflow-1.4.1/lib/python2.7/site-packages"
build --define with_jemalloc=true
build --define with_gcp_support=true
build --define with_hdfs_support=true
build --define with_s3_support=true
build --define with_xla_support=true
build --define with_gdr_support=true
build --define with_verbs_support=true
build --action_env TF_NEED_OPENCL="0"
build --action_env TF_NEED_CUDA="1"
build --action_env CUDA_TOOLKIT_PATH="/usr/local/cuda"
build --action_env TF_CUDA_VERSION="9.1"
build --action_env CUDNN_INSTALL_PATH="/usr/local/cuda-9.1"
build --action_env TF_CUDNN_VERSION="7"
build --action_env TF_CUDA_COMPUTE_CAPABILITIES="6.0,6.0,6.0,6.0,6.0,6.0,6.0,6.0"
build --action_env TF_CUDA_CLANG="0"
build --action_env GCC_HOST_COMPILER_PATH="/usr/bin/gcc"
build --config=cuda
test --config=cuda
build --define with_mpi_support=true
build:opt --cxxopt=-march=native --copt=-march=native
build:mkl --define using_mkl=true
build:mkl -c opt
build:mkl --copt="-DEIGEN_USE_VML"
build:monolithic --define framework_shared_object=false
build --define framework_shared_object=true

4.2 bazel 编译

bazel build -c opt --copt=-march="haswell" --config=cuda //tensorflow/tools/pip_package:build_pip_package

继续出错

INFO: Found 1 target...
ERROR: /root/vir-tensorflow-1.4.1/tensorflow-1.4.1/tensorflow/contrib/rnn/BUILD:217:1: error while parsing .d file: /root/.cache/bazel/_bazel_root/464f5f3c6697f75fdd1f3dea06540ee6/execroot/org_tensorflow/bazel-out/local_linux-opt/bin/tensorflow/contrib/rnn/_objs/python/ops/_gru_ops_gpu/tensorflow/contrib/rnn/kernels/gru_ops_gpu.cu.pic.d (No such file or directory).
In file included from external/eigen_archive/unsupported/Eigen/CXX11/Tensor:14:0,
                 from ./third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1,
                 from ./tensorflow/contrib/rnn/kernels/gru_ops.h:19,
                 from tensorflow/contrib/rnn/kernels/gru_ops_gpu.cu.cc:19:
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/Core:59:34: fatal error: math_functions.hpp: No such file or directory
compilation terminated.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.

参考：

https://github.com/tensorflow/tensorflow/issues/15573

https://stackoverflow.com/questions/43113508/math-functions-hpp-not-found-when-using-cuda-with-eigen/47807106#47807106

解决方法：

ln -s /usr/local/cuda/include/crt/math_functions.hpp /usr/local/cuda/include/math_functions.hpp

4.3 生成 whl 文件

bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg  

4.4 安装whl文件

pip install /tmp/tensorflow_pkg/tensorflow-1.4.1*