PyTorch 1.4 源码编译记录

0. 前言

• 目标：在 ubuntu 16.04 上编译PyTorch1.4源码。
• 参考资料：PyTorch 1.4 源码，release链接

1. 基本流程

• 参考资料：官方提供了教程
• 只提供了 conda 编译的流程，如果没有conda要从源码编译应该特别麻烦……
• 流程：
  • 第一步：安装依赖。
    • 公共依赖：conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi
    • 安装 mega：conda install -c pytorch magma-cuda90 # or [magma-cuda92 | magma-cuda100 | magma-cuda101 ] depending on your cuda version
  • 第二步：获取源码。
    • 获取pytorch源码：git clone --recursive https://github.com/pytorch/pytorch
    • 获取第三方库源码：git submodule sync && git submodule update --init --recursive
  • 第三步：编译源码。
    • Linux 中使用

export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py install

2. 常见问题

• git clone 的速度实在是太慢了……
  • 为了提高速度，花了半小时复制黏贴gitee上的repo，以及 thrid party 中的repo的一些submodules。
  • 在clone或者下载源码后，在pytorch目录下运行下面的命令下载源码，然后再执行 git submodule 命令。

git clone https://gitee.com/falaq/pybind11.git third_party/pybind11
git clone https://gitee.com/shoehall/cub.git third_party/cub
git clone https://gitee.com/qq2820/eigen-git-mirror.git third_party/eigen
git clone https://gitee.com/tianfuzeng/googletest.git third_party/googletest
git clone https://gitee.com/mirrors/google-benchmark.git third_party/benchmark
git clone https://gitee.com/cydiachen/protobuf.git third_party/protobuf
git clone https://gitee.com/falaq/ios-cmake.git third_party/ios-cmake
git clone https://gitee.com/madobet/NNPACK.git third_party/NNPACK
git clone https://gitee.com/falaq/gloo.git third_party/gloo
git clone https://gitee.com/falaq/pthreadpool.git third_party/pthreadpool
git clone https://gitee.com/falaq/FXdiv.git third_party/FXdiv
git clone https://gitee.com/falaq/FP16.git third_party/FP16
git clone https://gitee.com/falaq/psimd.git third_party/psimd
git clone https://gitee.com/falaq/zstd.git third_party/zstd
git clone https://gitee.com/falaq/cpuinfo.git third_party/cpuinfo
git clone https://gitee.com/falaq/enum34.git third_party/python-enum
git clone https://gitee.com/falaq/PeachPy.git third_party/python-peachpy
git clone https://gitee.com/falaq/six.git third_party/python-six
git clone https://gitee.com/mirrors/ONNX.git third_party/onnx
git clone https://gitee.com/falaq/onnx-tensorrt.git third_party/onnx-tensorrt
git clone https://gitee.com/falaq/sleef.git third_party/sleef
git clone https://gitee.com/falaq/ideep.git third_party/ideep
git clone https://gitee.com/mlb810/nccl.git third_party/nccl/nccl
git clone https://gitee.com/falaq/gemmlowp.git third_party/gemmlowp/gemmlowp
git clone https://gitee.com/mirrors/QNNPACK.git third_party/QNNPACK
git clone https://gitee.com/falaq/ARM_NEON_2_x86_SSE.git third_party/neon2sse
git clone https://gitee.com/guxiong007/fbgemm.git third_party/fbgemm
git clone https://gitee.com/falaq/foxi.git third_party/foxi
git clone https://gitee.com/falaq/tbb.git third_party/tbb
git clone https://gitee.com/falaq/fbjni.git android/libs/fbjni

# pybind11
git clone https://gitee.com/falaq/clang-cindex-python3.git third_party/pybind11/tools/clang

# fbgemm
git clone https://gitee.com/mirrors/AsmJit.git third_party/fbgemm/third_party/asmjit
git clone https://gitee.com/falaq/cpuinfo.git third_party/fbgemm/third_party/cpuinfo
git clone https://gitee.com/tianfuzeng/googletest.git third_party/fbgemm/third_party/googletest

# gloo
git clone https://gitee.com/tianfuzeng/googletest.git third_party/gloo/third-party/googletest

#ideep
git clone https://gitee.com/mirrors/MKL-DNN.git third_party/ideep/mkl-dnn

# onnx
git clone https://gitee.com/mirrors/google-benchmark.git third_party/onnx/third_party/benchmark
git clone https://gitee.com/falaq/pybind11.git third_party/onnx/third_party/pybind11
git clone https://gitee.com/falaq/clang-cindex-python3.git third_party/onnx/third_party/pybind11/tools/clang

# onnx-tensorrt
git clone https://gitee.com/mirrors/ONNX.git third_party/onnx-tensorrt/third_party/onnx
git clone https://gitee.com/mirrors/google-benchmark.git third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark
git clone https://gitee.com/falaq/pybind11.git third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11
git clone https://gitee.com/falaq/clang-cindex-python3.git third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang

# protobuf
git clone https://gitee.com/mirrors/google-benchmark.git third_party/protobuf/third_party/benchmark
git clone https://gitee.com/tianfuzeng/googletest.git third_party/protobuf/third_party/googletest