Ubuntu 22.04 x86_64 源码编译 pytorch-v2.0.1 笔记

依赖

cmake

sudo apt install cmake

cuda

Ubuntu 20.04 LTS 安装 nvidia 驱动 + cuda 11.8 从开始到放弃！_hkNaruto的博客-CSDN博客

源码

地址

GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration

https://github.com/pytorch/pytorch.git
git archive --format=tar --prefix=pytorch-v2.0.1/ v2.0.1 > /tmp/pytorch-v2.0.1.tar
tar -xvf pytorch-v2.0.1.tar 
cd pytorch-v2.0.1
make

故障处理

No CMAKE_CUDA_COMPILER could be found.

-- /usr/bin/c++ /tmp/pytorch-v2.0.1/torch/abi-check.cpp -o /tmp/pytorch-v2.0.1/build/abi-check
-- Determined _GLIBCXX_USE_CXX11_ABI=1
-- Could not find ccache. Consider installing ccache to speed up compilation.
-- Turning off deprecation warning due to glog.
-- Current compiler supports avx2 extension. Will build perfkernels.
-- Current compiler supports avx512f extension. Will build fbgemm.
-- The CUDA compiler identification is unknown
CMake Error at cmake/public/cuda.cmake:47 (enable_language):
  No CMAKE_CUDA_COMPILER could be found.

  Tell CMake where to find the compiler by setting either the environment
  variable "CUDACXX" or the CMake cache entry CMAKE_CUDA_COMPILER to the full
  path to the compiler, or to the compiler name if it is in the PATH.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:43 (include)
  CMakeLists.txt:717 (include)


-- Configuring incomplete, errors occurred!
See also "/tmp/pytorch-v2.0.1/build/CMakeFiles/CMakeOutput.log".
See also "/tmp/pytorch-v2.0.1/build/CMakeFiles/CMakeError.log".
make: *** [Makefile:6：all] 错误 1

解决

export PATH=$PATH:/usr/local/cuda/bin/

does not comtain a CMakeLists.txt file

CMake Error at cmake/Dependencies.cmake:434 (add_subdirectory):
  The source directory

    /tmp/pytorch-v2.0.1/third_party/pthreadpool

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:437 (set_property):
  set_property could not find TARGET pthreadpool.  Perhaps it has not yet
  been created.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:477 (add_subdirectory):
  The source directory

    /tmp/pytorch-v2.0.1/third_party/cpuinfo

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:482 (set_property):
  set_property could not find TARGET cpuinfo.  Perhaps it has not yet been
  created.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:504 (add_subdirectory):
  The source directory

    /tmp/pytorch-v2.0.1/third_party/QNNPACK

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:510 (target_compile_options):
  Cannot specify compile options for target "qnnpack" which is not built by
  this project.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:515 (set_property):
  set_property could not find TARGET qnnpack.  Perhaps it has not yet been
  created.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:516 (set_property):
  set_property could not find TARGET cpuinfo.  Perhaps it has not yet been
  created.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
CMake Error at aten/src/ATen/native/quantized/cpu/qnnpack/CMakeLists.txt:330 (add_subdirectory):
  add_subdirectory given source
  "/tmp/pytorch-v2.0.1/third_party/cpuinfo/deps/clog" which is not an
  existing directory.


CMake Error at aten/src/ATen/native/quantized/cpu/qnnpack/CMakeLists.txt:334 (set_property):
  set_property could not find TARGET clog.  Perhaps it has not yet been
  created.


CMake Error at aten/src/ATen/native/quantized/cpu/qnnpack/CMakeLists.txt:352 (add_subdirectory):
  The binary directory

    /tmp/pytorch-v2.0.1/build/confu-deps/cpuinfo

  is already used to build a source directory.  It cannot be used to build
  source directory

    /tmp/pytorch-v2.0.1/third_party/cpuinfo

  Specify a unique binary directory name.


CMake Error at aten/src/ATen/native/quantized/cpu/qnnpack/CMakeLists.txt:362 (add_subdirectory):
  The binary directory

    /tmp/pytorch-v2.0.1/build/confu-deps/pthreadpool

  is already used to build a source directory.  It cannot be used to build
  source directory

    /tmp/pytorch-v2.0.1/third_party/pthreadpool

  Specify a unique binary directory name.


CMake Error at aten/src/ATen/native/quantized/cpu/qnnpack/CMakeLists.txt:388 (add_subdirectory):
  The source directory

    /tmp/pytorch-v2.0.1/third_party/FXdiv

  does not contain a CMakeLists.txt file.


CMake Error at aten/src/ATen/native/quantized/cpu/qnnpack/CMakeLists.txt:403 (add_subdirectory):
  The source directory

    /tmp/pytorch-v2.0.1/third_party/psimd

  does not contain a CMakeLists.txt file.


CMake Error at aten/src/ATen/native/quantized/cpu/qnnpack/CMakeLists.txt:420 (add_subdirectory):
  The source directory

    /tmp/pytorch-v2.0.1/third_party/FP16

  does not contain a CMakeLists.txt file.


CMake Error at cmake/Dependencies.cmake:571 (set_property):
  set_property could not find TARGET cpuinfo.  Perhaps it has not yet been
  created.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


-- Brace yourself, we are building NNPACK
CMake Error at cmake/External/nnpack.cmake:71 (add_subdirectory):
  The source directory

    /tmp/pytorch-v2.0.1/third_party/NNPACK

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:601 (include)
  CMakeLists.txt:717 (include)


CMake Error at cmake/External/nnpack.cmake:76 (set_property):
  set_property could not find TARGET nnpack.  Perhaps it has not yet been
  created.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:601 (include)
  CMakeLists.txt:717 (include)


CMake Error at cmake/External/nnpack.cmake:77 (set_property):
  set_property could not find TARGET pthreadpool.  Perhaps it has not yet
  been created.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:601 (include)
  CMakeLists.txt:717 (include)


CMake Error at cmake/External/nnpack.cmake:78 (set_property):
  set_property could not find TARGET cpuinfo.  Perhaps it has not yet been
  created.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:601 (include)
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:642 (add_subdirectory):
  The source directory

    /tmp/pytorch-v2.0.1/third_party/XNNPACK

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:832 (add_subdirectory):
  The source directory

    /tmp/pytorch-v2.0.1/third_party/fbgemm

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:833 (set_property):
  set_property could not find TARGET fbgemm_generic.  Perhaps it has not yet
  been created.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:834 (set_property):
  set_property could not find TARGET fbgemm_avx2.  Perhaps it has not yet
  been created.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:835 (set_property):
  set_property could not find TARGET fbgemm_avx512.  Perhaps it has not yet
  been created.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:836 (set_property):
  set_property could not find TARGET fbgemm.  Perhaps it has not yet been
  created.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


-- Could NOT find Numa (missing: Numa_INCLUDE_DIR Numa_LIBRARIES) 
CMake Warning at cmake/Dependencies.cmake:902 (message):
  Not compiling with NUMA.  Suppress this warning with -DUSE_NUMA=OFF
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Warning at cmake/Dependencies.cmake:981 (message):
  Not compiling with ITT.  Suppress this warning with -DUSE_ITT=OFF
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


CMake Error at cmake/public/utils.cmake:188 (set):
  set given invalid arguments for CACHE mode.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:983 (caffe2_update_option)
  CMakeLists.txt:717 (include)


CMake Error at cmake/Dependencies.cmake:1000 (add_subdirectory):
  The source directory

    /tmp/pytorch-v2.0.1/third_party/FP16

  does not contain a CMakeLists.txt file.
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


-- Using third party subdirectory Eigen.
-- Setting Python's library to /usr/lib/python3.10
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.10.12", minimum required is "3.0") 
-- Found PythonLibs: /usr/lib/python3.10 (Required is at least version "3.0") 
CMake Error at cmake/Dependencies.cmake:1088 (if):
  if given arguments:

    "VERSION_LESS" "3"

  Unknown arguments specified
Call Stack (most recent call first):
  CMakeLists.txt:717 (include)


-- Configuring incomplete, errors occurred!
See also "/tmp/pytorch-v2.0.1/build/CMakeFiles/CMakeOutput.log".
See also "/tmp/pytorch-v2.0.1/build/CMakeFiles/CMakeError.log".
make: *** [Makefile:6：all] 错误 1

切换到git 源码目录重试

cd ~/Downloads/src/pytorch/
git checkout v2.0.1
git submodule init
git submodule update

注意，调整下eigen地址

[submodule "third_party/eigen"]
    ignore = dirty
    path = third_party/eigen
    #url = https://gitlab.com/libeigen/eigen.git
    url = https://gitee.com/masa-laboratory/eigen

根据以往stable diffusion webui经验，采用conda创建python3.10.6

conda create -n pytorch-build python=3.10.6
conda activate pytorch-build

No module named 'yaml'

ModuleNotFoundError: No module named 'yaml'
CMake Error at cmake/Codegen.cmake:197 (message):
  Failed to get generated_headers list
Call Stack (most recent call first):
  caffe2/CMakeLists.txt:2 (include)

解决

pip install pyyaml

target specific option mismatch

error: inlining failed in call to ‘always_inline’ ‘_mm512_set_epi8’: target specific option mismatch
  121 | _mm512_set_epi8 (char __q63, char __q62, char __q61, char __q60,
      | ^~~~~~~~~~~~~~~

估计是当前gcc版本过高

gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04) 

待续。。。

20230831继续：

当前状态

故障