Pytorch 在 Ubuntu18.04 上的编译、安装、与问题

左手的ㄟ右手 2022-03-26 07:58 479阅读 0赞

这里主要是develop开发版的编译与安装。如果不是开发版，安装是非常容易的，anaconda已经做得非常到位了，可参考该帖的后面部分,  
[https://blog.csdn.net/tanmx219/article/details/82831964][https_blog.csdn.net_tanmx219_article_details_82831964]

在自己编译的情况下，和官网相比，我每次重新编译总是缺些什么，所以决定记录一下，

[https://github.com/pytorch/pytorch][https_github.com_pytorch_pytorch]  
[https://m.oldpan.me/archives/pytorch-build-simple-instruction][https_m.oldpan.me_archives_pytorch-build-simple-instruction]

目前是pytorch v1.0  
\------------------------------------------------------------------------------------

conda install numpy pyyaml mkl mkl-include setuptools cmake cffi typing pybind11
    conda install cudnn (cudatoolkit: 9.2-0+cudnn:7.2.1-cuda9.2_0)
    conda install ninja (optional, warning: C++ will not be compiled incrementally)
    conda install -c pytorch magma-cuda92 (optional, Add LAPACK support for the GPU if needed)
    git clone --recursive https://github.com/pytorch/pytorch
    cd pytorch
    
    export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
    如果不需要使用cuda的话，这里还要加上一句：export NO_CUDA=1
    
    python setup.py install 
    对于开发者模式，可以使用
    python setup.py build develop

\------------------------------------------------------------------------------------

源码安装的Pytorch，卸载需要执行：

pip uninstall torch
    python setup.py clean

\------------------------------------------------------------------------------------

注意git clone之前确保已经安装git，如果没有安装，sudo apt-get install git-all安装一下。

### 碰到的问题 ###

Linux的问题总是千奇百怪，每次都不一样，也记录一下

=========================================================================

Problem 1:

\---------------------------------------------------------------------------------------------------------------------------------

Cuda的安装，网上讲的安装方法非常多，其中包括一种比较有意思的，  
[https://askubuntu.com/questions/1028830/how-do-i-install-cuda-on-ubuntu-18-04][https_askubuntu.com_questions_1028830_how-do-i-install-cuda-on-ubuntu-18-04]

-----how to install-----
    sudo add-apt-repository ppa:graphics-drivers/ppa
    sudo apt update
    sudo ubuntu-drivers autoinstall
    reboot
    sudo apt install nvidia-cuda-toolkit gcc-6
    nvcc --version
    
    -----replace------
    sudo sh cuda_9.1.85_387.26_linux.run (--override)
    to uninstall the NVIDIA Driver, run nvidia-uninstall
    
    -----how to remove-----
    sudo apt-get remove --purge nvidia-*
    sudo apt-get remove --purge libnvidia-*

这个方法是通过PPA proprietary graphics driver team 开发的包安装，可参考：  
[https://launchpad.net/~graphics-drivers/+archive/ubuntu/ppa][https_launchpad.net_graphics-drivers_archive_ubuntu_ppa]  
这里有其团队的信息，  
[https://launchpad.net/~graphics-drivers][https_launchpad.net_graphics-drivers]  
不过写这东东时我也是第一次用这种方式安装，检查了一下，貌似通过这种形式安装的都是最新版本，还没看到过详细文档。  
官网上提到：  
Current long-lived branch release: \`nvidia-410\` (410.66)  
Dropped support for Fermi series ([https://nvidia.custhelp.com/app/answers/detail/a\_id/4656][https_nvidia.custhelp.com_app_answers_detail_a_id_4656])  
Old long-lived branch release: \`nvidia-390\` (390.87)  
默认是nvidia-410,  
总之，通过这种方式会安装很多包，这里装了多少多余的东西俺就不知道了。

\--------------------

那假设我就是想安装nvidia-390怎么办呢？还是老办法保险：  
sudo apt-get install nvidia-driver-390  
sudo apt-get install nvidia-driver-390-dev

=========================================================================

Problem 2:

\---------------------------------------------------------------------------------------------------------------------------------

CMake Error at cmake/public/cuda.cmake:318 (message):  
  CUDA 9.1 is not compatible with std::tuple from GCC version >= 6.  Please  
  upgrade to CUDA 9.2 or use the following option to use another version (for  
  example):  
    -DCUDA\_HOST\_COMPILER=/usr/bin/gcc-5  
.....  
\-- Configuring incomplete, errors occurred!  
See also "/home/matthew/dev/pytorch/build/CMakeFiles/CMakeOutput.log".  
See also "/home/matthew/dev/pytorch/build/CMakeFiles/CMakeError.log".  
Failed to run 'bash ../tools/build\_pytorch\_libs.sh --use-cuda --use-fbgemm --use-nnpack --use-mkldnn --use-qnnpack caffe2'

网上有不少解决办法，参考：[https://github.com/pytorch/pytorch/issues/14152][https_github.com_pytorch_pytorch_issues_14152]

Solution 1:

\--------------------------------------------------------------------------------------------------------------------------------

版本不对，大概是先卸载再安装，参考：

step1: remove the previous cuda
    sudo apt-get --purge remove nvidia-cuda-toolkit
    conda remove cudnn (if has installed a different version needed)
    
    step2: install cudnn  (this will install cudnn7_cuda9.2）
    conda install cudnn

[https://blog.csdn.net/tanmx219/article/details/86210023][https_blog.csdn.net_tanmx219_article_details_86210023]  
[https://github.com/pytorch/pytorch/issues/14152][https_github.com_pytorch_pytorch_issues_14152]

=========================================================================

Problem 3:

\---------------------------------------------------------------------------------------------------------------------------------

$ python setup.py build develop  
Building wheel torch-1.1.0a0+964732f  
running build  
running build\_deps  
setup.py::build\_deps::run()  
\+ SYNC\_COMMAND=cp  
......  
\--     USE\_GLOO\_IBVERBS    : OFF  
\--   Public Dependencies  : Threads::Threads;caffe2::mkl;caffe2::mkldnn  
\--   Private Dependencies : qnnpack;nnpack;cpuinfo;fbgemm;fp16;gloo;aten\_op\_header\_gen;onnxifi\_loader;rt;gcc\_s;gcc;dl  
\-- Configuring done  
CMake Error in caffe2/CMakeLists.txt:  
  Imported target "caffe2::mkl" **includes non-existent path**  
    "/home/matthew/anaconda3/envs/torchdev/include"  
  in its INTERFACE\_INCLUDE\_DIRECTORIES.  Possible reasons include:  
  \* The path was deleted, renamed, or moved to another location.  
  \* An install or uninstall procedure did not complete successfully.  
  \* The installation package was faulty and references files it does not  
  provide.

\-- Generating done  
\-- Build files have been written to: /home/matthew/dev/pytorch/build  
Failed to run 'bash ../tools/build\_pytorch\_libs.sh --use-fbgemm --use-nnpack --use-mkldnn --use-qnnpack caffe2'

解决办法  
\---------------------------------------------------------------------------------------------------------------------------------

这个问题是因为上一次编译没编译完，这次接着编译，但命令路径变了，所以必须清空重来  
python setup.py clean  
python setup.py build develop

=========================================================================

Problem 3:

\--------------------------------------------------------------------------------------------------------------------------------

The Nouveau kernel driver is currently in use by your system.  This driver is incompatible with the NVIDIA driver,

参考：  
[https://tutorials.technology/tutorials/85-How-to-remove-Nouveau-kernel-driver-Nvidia-install-error.html][https_tutorials.technology_tutorials_85-How-to-remove-Nouveau-kernel-driver-Nvidia-install-error.html]  
How to remove Nouveau kernel driver (fix Nvidia install error)

sudo apt-get remove nvidia*
    sudo apt autoremove
    sudo apt-get install dkms build-essential linux-headers-generic
    sudo vim /etc/modprobe.d/blacklist.conf
    
    blacklist nouveau
    blacklist lbm-nouveau
    options nouveau modeset=0
    alias nouveau off
    alias lbm-nouveau off
    
    echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
    sudo update-initramfs -u
    reboot

sudo apt-get --purge remove xserver-xorg-video-nouveau

=========================================================================

Problem 4:

\--------------------------------------------------------------------------------------------------------------------------------

\-- Caffe2: Cannot find glog automatically. Using legacy find.  
\-- Could NOT find glog (missing: GLOG\_INCLUDE\_DIR GLOG\_LIBRARY)   
CMake Warning at cmake/public/glog.cmake:66 (message):  
  Caffe2: glog cannot be found.  Depending on whether you are building Caffe2  
  or a Caffe2 dependent library, the next warning / error will give you more  
  info.  
Call Stack (most recent call first):  
  cmake/Dependencies.cmake:291 (include)  
  CMakeLists.txt:219 (include)

===>>>

$ sudo apt-get install libgflags-dev    libgoogle-glog-dev

=========================================================================

Problem 5:

\--------------------------------------------------------------------------------------------------------------------------------

\-- Found CUDA: /usr/local/cuda (found version "10.0")   
\-- Caffe2: CUDA detected: 10.0  
\-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc  
\-- Caffe2: CUDA toolkit directory: /usr/local/cuda  
\-- Caffe2: Header version is: 10.0  
\-- Could NOT find CUDNN (missing: CUDNN\_INCLUDE\_DIR CUDNN\_LIBRARY)   
CMake Warning at cmake/public/cuda.cmake:101 (message):  
  Caffe2: Cannot find cuDNN library.  Turning the option off  
Call Stack (most recent call first):  
  cmake/Dependencies.cmake:685 (include)  
  CMakeLists.txt:219 (include)

===>>>

安装了cuda+cudnn后这个就会消失

[https_blog.csdn.net_tanmx219_article_details_82831964]: https://blog.csdn.net/tanmx219/article/details/82831964
[https_github.com_pytorch_pytorch]: https://github.com/pytorch/pytorch
[https_m.oldpan.me_archives_pytorch-build-simple-instruction]: https://m.oldpan.me/archives/pytorch-build-simple-instruction
[https_askubuntu.com_questions_1028830_how-do-i-install-cuda-on-ubuntu-18-04]: https://askubuntu.com/questions/1028830/how-do-i-install-cuda-on-ubuntu-18-04
[https_launchpad.net_graphics-drivers_archive_ubuntu_ppa]: https://launchpad.net/~graphics-drivers/+archive/ubuntu/ppa
[https_launchpad.net_graphics-drivers]: https://launchpad.net/~graphics-drivers
[https_nvidia.custhelp.com_app_answers_detail_a_id_4656]: https://nvidia.custhelp.com/app/answers/detail/a_id/4656
[https_github.com_pytorch_pytorch_issues_14152]: https://github.com/pytorch/pytorch/issues/14152
[https_blog.csdn.net_tanmx219_article_details_86210023]: https://blog.csdn.net/tanmx219/article/details/86210023
[https_tutorials.technology_tutorials_85-How-to-remove-Nouveau-kernel-driver-Nvidia-install-error.html]: https://tutorials.technology/tutorials/85-How-to-remove-Nouveau-kernel-driver-Nvidia-install-error.html