ubuntu16.04+anaconda3+python2.7虚拟环境+caffe配置过程及踩坑经历

最新推荐文章于 2021-09-30 18:51:07 发布

zxmyoung

最新推荐文章于 2021-09-30 18:51:07 发布

阅读量1.2k

点赞数

分类专栏： caffe

本文链接：https://blog.csdn.net/zxmyoung/article/details/107527002

版权

caffe 专栏收录该内容

14 篇文章 0 订阅

订阅专栏

. 前提:

安装前服务器情况,已经安装好了:

CUDNN=7.3.0
CUDA=10.0.130
Opencv 2.4.13

相应命令为:

cuda 版本
cat /usr/local/cuda/version.txt
cudnn 版本
cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
opencv版本
pkg-config opencv --modversion

-----------------------------------------以上是别人的前提条件，我的电脑这些都已安装好，下面是我的前提环境

准备caffe, python环境

从https://github.com/BVLC/caffe下载caffe, 用unzip命令解压

python我使用的是Anaconda3虚拟环境, 最后附上environment_caffe_27.yml文件, 如果需要可以直接使用下面的命令安装,或者手动安装.

conda env create -f environment_caffe_27.yml

我是手动在caffe文件下创建的

具体内容如下：我创建的虚拟环境名称是caffe_27, python2.7

name: caffe_27
channels:
  - defaults
dependencies:
  - backports=1.0=py27_1
  - backports.functools_lru_cache=1.5=py27_1
  - backports.shutil_get_terminal_size=1.0.0=py27_2
  - backports_abc=0.5=py27_0
  - blas=1.0=mkl
  - ca-certificates=2019.1.23=0
  - certifi=2019.3.9=py27_0
  - cloudpickle=0.8.0=py27_0
  - cycler=0.10.0=py27_0
  - cytoolz=0.9.0.1=py27h14c3975_1
  - dask-core=1.1.5=py_0
  - dbus=1.13.6=h746ee38_0
  - decorator=4.4.0=py27_1
  - enum34=1.1.6=py27_1
  - expat=2.2.6=he6710b0_0
  - fontconfig=2.13.0=h9420a91_0
  - freetype=2.9.1=h8a8886c_1
  - functools32=3.2.3.2=py27_1
  - futures=3.2.0=py27_0
  - glib=2.56.2=hd408876_0
  - gst-plugins-base=1.14.0=hbbd80ab_1
  - gstreamer=1.14.0=hb453b48_1
  - h5py=2.9.0=py27h7918eee_0
  - hdf5=1.10.4=hb1b8bf9_0
  - icu=58.2=h9c2bf20_1
  - imageio=2.5.0=py27_0
  - intel-openmp=2019.3=199
  - ipython=5.8.0=py27_0
  - ipython_genutils=0.2.0=py27_0
  - jpeg=9b=h024ee3a_2
  - kiwisolver=1.0.1=py27hf484d3e_0
  - leveldb=1.20=hf484d3e_1
  - libedit=3.1.20181209=hc058e9b_0
  - libffi=3.2.1=hd88cf55_4
  - libgcc-ng=8.2.0=hdf63c60_1
  - libgfortran-ng=7.3.0=hdf63c60_0
  - libpng=1.6.36=hbc83047_0
  - libprotobuf=3.2.0=0
  - libstdcxx-ng=8.2.0=hdf63c60_1
  - libtiff=4.0.10=h2733197_2
  - libuuid=1.0.3=h1bed415_2
  - libxcb=1.13=h1bed415_1
  - libxml2=2.9.9=he19cac6_0
  - linecache2=1.0.0=py27_0
  - matplotlib=2.2.3=py27hb69df0a_0
  - mkl=2019.3=199
  - mkl_fft=1.0.10=py27ha843d7b_0
  - mkl_random=1.0.2=py27hd81dba3_0
  - ncurses=6.1=he6710b0_1
  - networkx=2.2=py27_1
  - nose=1.3.7=py27_2
  - numpy=1.16.2=py27h7e9f1db_0
  - numpy-base=1.16.2=py27hde5b4d6_0
  - olefile=0.46=py27_0
  - openssl=1.1.1b=h7b6447c_1
  - pandas=0.24.2=py27he6710b0_0
  - pathlib2=2.3.3=py27_0
  - pcre=8.43=he6710b0_0
  - pexpect=4.6.0=py27_0
  - pickleshare=0.7.5=py27_0
  - pillow=5.4.1=py27h34e0f95_0
  - pip=19.0.3=py27_0
  - prompt_toolkit=1.0.15=py27_0
  - protobuf=3.2.0=py27_0
  - ptyprocess=0.6.0=py27_0
  - pygments=2.3.1=py27_0
  - pyparsing=2.3.1=py27_0
  - pyqt=5.9.2=py27h05f1152_2
  - python=2.7.16=h9bab390_0
  - python-dateutil=2.8.0=py27_0
  - python-gflags=3.1.2=py27_0
  - pytz=2018.9=py27_0
  - pywavelets=1.0.2=py27hdd07704_0
  - pyyaml=5.1=py27h7b6447c_0
  - qt=5.9.7=h5867ecd_1
  - readline=7.0=h7b6447c_5
  - scandir=1.10.0=py27h7b6447c_0
  - scikit-image=0.14.2=py27he6710b0_0
  - scipy=1.2.1=py27h7c811a0_0
  - setuptools=40.8.0=py27_0
  - simplegeneric=0.8.1=py27_2
  - singledispatch=3.4.0.3=py27_0
  - sip=4.19.8=py27hf484d3e_0
  - six=1.12.0=py27_0
  - snappy=1.1.7=hbae5bb6_3
  - sqlite=3.27.2=h7b6447c_0
  - subprocess32=3.5.3=py27h7b6447c_0
  - tk=8.6.8=hbc83047_0
  - toolz=0.9.0=py27_0
  - tornado=5.1.1=py27h7b6447c_0
  - traceback2=1.4.0=py27_0
  - traitlets=4.3.2=py27_0
  - unittest2=1.1.0=py27_0
  - wcwidth=0.1.7=py27_0
  - wheel=0.33.1=py27_0
  - xz=5.2.4=h14c3975_4
  - yaml=0.1.7=had09818_2
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.3.7=h0b5b093_0
  - pip:
    - cython==0.29.6
    - dask==1.1.5
prefix: /home/zhai/anaconda3/envs/caffe_27

接下来就是在caffe路径下，打开命令界面，并进入刚才创建的虚拟环境caffe_27并激活

然后在caffe路径下有requirements.txt。执行以下命令，安装各种安装包

pip install -r python/requirements.txt

注意：

protobuf版本在requirements.txt中值写了最小值,实测最新版的protobuf不知此, 建议安装3.2版本, 即使用conda install protobuf=3.2。我没有操作这一步，也成功了。
requirements.txt中,要求python-dateutil>=1.4,<2, 但是这个版本太老了找不到, 我安装的了2.8版本,实测编译通过,调用正常.我也是直接执行上面的命令，后面没有报错。

复制修改Makefileconfig文件，修改Make文件

cp Makefile.config.example Makefile.config

复制后，并修改

下面是我修改后的，具体情况根据自身修改（需要改动的地方我都用超多感叹好标识）

## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1  #！！！！！！！！！！！！！！！！！！！！！

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0
# This code is taken from https://github.com/sh1r0/caffe-android-lib
# USE_HDF5 := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#	You should not set this flag if you will be reading LMDBs with any
#	possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3，我的版本是2
OPENCV_VERSION := 2  #！！！！！！！！！！！！！！！！

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_50,code=sm_50 \
		-gencode arch=compute_52,code=sm_52 \
		-gencode arch=compute_60,code=sm_60 \
		-gencode arch=compute_61,code=sm_61 \
		-gencode arch=compute_61,code=compute_61
# ！！！！！！！！！！！！！！！！！！！！！！！！
# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := open #！！！！！！！！！！！！！！！！！！！
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
#PYTHON_INCLUDE := /usr/include/python2.7 \
		#/usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
ANACONDA_HOME := $(HOME)/anaconda3/envs/caffe_27
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
		  $(ANACONDA_HOME)/include/python2.7 \
		  $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include
#！！！！！！！！！！！！！！！！！！！！！！！！以上三行都要根据自己的路径进行更换并取消注释
# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python3 python3.5m
# PYTHON_INCLUDE := /usr/include/python3.5m \
#                 /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
# PYTHON_LIB := /usr/lib
PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial
# ！！！！！！！！！！！！！！！！！！！以上两行根据我的修改
# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @

修改Makefile文件，下面是我修改后的。

修改 caffe 目录下的 Makefile 文件（这两个不改都会报错，编译类型错误等等。）

将：
LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_hl hdf5 
改为：
LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl hdf5_serial

将：
 NVCCFLAGS +=-ccbin=$(CXX) -Xcompiler-fPIC $(COMMON_FLAGS) 
替换为：
 NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)

makefile上面两个可以不改，但是下面这个的改，亲测，不改会出错，因为编译时会出现下面的错误

ImportError: No module named google.protobuf.internal

解决方法：

首先查看 protobuf 信息

whereis protoc  #查看那些路径下安装了protobuf
which protoc  #查看默认选用的protobuf
protoc --version  #查看当前默认的protobuf的版本（配置caffe时很重要

我一开始改为我的虚拟环境caffe_27下的protoc，失败，还是老老实实的改为系统环境的，问题解决了（但是就是不知道为啥改为自己虚拟环境的就不行）

接下来虚拟环境添加到环境变量，不然报错

gedit ~/.bashrc

export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH（注意：path/to是自己caffe的路径），如下我的

export PYTHONPATH=/home/zhai/caffe/python:$PYTHONPATH

保存，生效

source ~/.bashrc

开始编译

make all –j8
make test –j8
make runtest –j8
make pycaffe  -j8

我是把能避免的坑到都考虑了，这里尽量没有坑。按照我的这个步骤现在是没坑的，放一个编译测试成功的图

当执行最后一个命令时，我的又出现问题了

Makefile:517: recipe for target 'python/caffe/_caffe.so' failed
make: *** [python/caffe/_caffe.so] Error 1

查了一下原因是Makefile.config里python路径没有配置对
解决方法：
1.打开终端，输入python，输入以下命令得到python2.7的地址

2.在Makefile.config中找到PYTHON_INCLUDE := /usr/include/python2.7 \这一行，修改下面一行的地址为刚刚得到的地址，再次编译即可。说白了，就是由于我的粗心，把这个位置的三行都要取消注释，我只取消一行，太粗心了。