Pytorch 在 Ubuntu18.04 上的编译、安装、与问题

这里主要是develop开发版的编译与安装。如果不是开发版，安装是非常容易的，anaconda已经做得非常到位了，可参考该帖的后面部分,
https://blog.csdn.net/tanmx219/article/details/82831964

在自己编译的情况下，和官网相比，我每次重新编译总是缺些什么，所以决定记录一下，

https://github.com/pytorch/pytorch
https://m.oldpan.me/archives/pytorch-build-simple-instruction

目前是pytorch v1.0
------------------------------------------------------------------------------------

conda install numpy pyyaml mkl mkl-include setuptools cmake cffi typing pybind11
conda install cudnn (cudatoolkit: 9.2-0+cudnn:7.2.1-cuda9.2_0)
conda install ninja (optional, warning: C++ will not be compiled incrementally)
conda install -c pytorch magma-cuda92 (optional, Add LAPACK support for the GPU if needed)
git clone --recursive https://github.com/pytorch/pytorch
cd pytorch

export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
如果不需要使用cuda的话，这里还要加上一句：export NO_CUDA=1

python setup.py install 
对于开发者模式，可以使用
python setup.py build develop

------------------------------------------------------------------------------------

源码安装的Pytorch，卸载需要执行：

pip uninstall torch
python setup.py clean

------------------------------------------------------------------------------------

注意git clone之前确保已经安装git，如果没有安装，sudo apt-get install git-all安装一下。

 

碰到的问题

Linux的问题总是千奇百怪，每次都不一样，也记录一下

=========================================================================

Problem 1:

---------------------------------------------------------------------------------------------------------------------------------

Cuda的安装，网上讲的安装方法非常多，其中包括一种比较有意思的，
https://askubuntu.com/questions/1028830/how-do-i-install-cuda-on-ubuntu-18-04

-----how to install-----
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo ubuntu-drivers autoinstall
reboot
sudo apt install nvidia-cuda-toolkit gcc-6
nvcc --version

-----replace------
sudo sh cuda_9.1.85_387.26_linux.run (--override)
to uninstall the NVIDIA Driver, run nvidia-uninstall

-----how to remove-----
sudo apt-get remove --purge nvidia-*
sudo apt-get remove --purge libnvidia-*

这个方法是通过PPA proprietary graphics driver team 开发的包安装，可参考：
https://launchpad.net/~graphics-drivers/+archive/ubuntu/ppa
这里有其团队的信息，
https://launchpad.net/~graphics-drivers
不过写这东东时我也是第一次用这种方式安装，检查了一下，貌似通过这种形式安装的都是最新版本，还没看到过详细文档。
官网上提到：
Current long-lived branch release: `nvidia-410` (410.66)
Dropped support for Fermi series (https://nvidia.custhelp.com/app/answers/detail/a_id/4656)
Old long-lived branch release: `nvidia-390` (390.87)
默认是nvidia-410,
总之，通过这种方式会安装很多包，这里装了多少多余的东西俺就不知道了。

--------------------

那假设我就是想安装nvidia-390怎么办呢？还是老办法保险：
sudo apt-get install nvidia-driver-390
sudo apt-get install nvidia-driver-390-dev

=========================================================================

Problem 2:

---------------------------------------------------------------------------------------------------------------------------------

CMake Error at cmake/public/cuda.cmake:318 (message):
  CUDA 9.1 is not compatible with std::tuple from GCC version >= 6.  Please
  upgrade to CUDA 9.2 or use the following option to use another version (for
  example):
    -DCUDA_HOST_COMPILER=/usr/bin/gcc-5
.....
-- Configuring incomplete, errors occurred!
See also "/home/matthew/dev/pytorch/build/CMakeFiles/CMakeOutput.log".
See also "/home/matthew/dev/pytorch/build/CMakeFiles/CMakeError.log".
Failed to run 'bash ../tools/build_pytorch_libs.sh --use-cuda --use-fbgemm --use-nnpack --use-mkldnn --use-qnnpack caffe2'

网上有不少解决办法，参考：https://github.com/pytorch/pytorch/issues/14152

Solution 1:

--------------------------------------------------------------------------------------------------------------------------------

版本不对，大概是先卸载再安装，参考：

step1: remove the previous cuda
sudo apt-get --purge remove nvidia-cuda-toolkit
conda remove cudnn (if has installed a different version needed)

step2: install cudnn  (this will install cudnn7_cuda9.2）
conda install cudnn

https://blog.csdn.net/tanmx219/article/details/86210023
https://github.com/pytorch/pytorch/issues/14152

=========================================================================

Problem 3:

---------------------------------------------------------------------------------------------------------------------------------

$ python setup.py build develop
Building wheel torch-1.1.0a0+964732f
running build
running build_deps
setup.py::build_deps::run()
+ SYNC_COMMAND=cp
......
--     USE_GLOO_IBVERBS    : OFF
--   Public Dependencies  : Threads::Threads;caffe2::mkl;caffe2::mkldnn
--   Private Dependencies : qnnpack;nnpack;cpuinfo;fbgemm;fp16;gloo;aten_op_header_gen;onnxifi_loader;rt;gcc_s;gcc;dl
-- Configuring done
CMake Error in caffe2/CMakeLists.txt:
  Imported target "caffe2::mkl" includes non-existent path
    "/home/matthew/anaconda3/envs/torchdev/include"
  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:
  * The path was deleted, renamed, or moved to another location.
  * An install or uninstall procedure did not complete successfully.
  * The installation package was faulty and references files it does not
  provide.

-- Generating done
-- Build files have been written to: /home/matthew/dev/pytorch/build
Failed to run 'bash ../tools/build_pytorch_libs.sh --use-fbgemm --use-nnpack --use-mkldnn --use-qnnpack caffe2'

解决办法
---------------------------------------------------------------------------------------------------------------------------------

这个问题是因为上一次编译没编译完，这次接着编译，但命令路径变了，所以必须清空重来
python setup.py clean
python setup.py build develop

=========================================================================

Problem 3:

--------------------------------------------------------------------------------------------------------------------------------

The Nouveau kernel driver is currently in use by your system.  This driver is incompatible with the NVIDIA driver,

参考：
https://tutorials.technology/tutorials/85-How-to-remove-Nouveau-kernel-driver-Nvidia-install-error.html
How to remove Nouveau kernel driver (fix Nvidia install error)

sudo apt-get remove nvidia*
sudo apt autoremove
sudo apt-get install dkms build-essential linux-headers-generic
sudo vim /etc/modprobe.d/blacklist.conf

blacklist nouveau
blacklist lbm-nouveau
options nouveau modeset=0
alias nouveau off
alias lbm-nouveau off

echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
sudo update-initramfs -u
reboot

sudo apt-get --purge remove xserver-xorg-video-nouveau

=========================================================================

Problem 4:

--------------------------------------------------------------------------------------------------------------------------------

-- Caffe2: Cannot find glog automatically. Using legacy find.
-- Could NOT find glog (missing: GLOG_INCLUDE_DIR GLOG_LIBRARY) 
CMake Warning at cmake/public/glog.cmake:66 (message):
  Caffe2: glog cannot be found.  Depending on whether you are building Caffe2
  or a Caffe2 dependent library, the next warning / error will give you more
  info.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:291 (include)
  CMakeLists.txt:219 (include)

===>>>

$ sudo apt-get install libgflags-dev    libgoogle-glog-dev

=========================================================================

Problem 5:

--------------------------------------------------------------------------------------------------------------------------------

-- Found CUDA: /usr/local/cuda (found version "10.0") 
-- Caffe2: CUDA detected: 10.0
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 10.0
-- Could NOT find CUDNN (missing: CUDNN_INCLUDE_DIR CUDNN_LIBRARY) 
CMake Warning at cmake/public/cuda.cmake:101 (message):
  Caffe2: Cannot find cuDNN library.  Turning the option off
Call Stack (most recent call first):
  cmake/Dependencies.cmake:685 (include)
  CMakeLists.txt:219 (include)

===>>> 

安装了cuda+cudnn后这个就会消失