【caffe源码研究】第一章：安装篇(上): jumbo版

下载caffe

下载caffe的源码

[fangjin@caffe ~]$ git clone https://github.com/BVLC/caffe.git 

安装依赖库

caffe涉及到一些依赖库，先使用jumbo安装。

[fangjin@caffe ~]$ jumbo  install pkg-config snappy  opencv hdf5  google-gflags google-glog  leveldb protobuf

这里为了方便统一将其他依赖库安装到/home/users/fangjin/opt目录下。更合理的做法是为不同的依赖库建立不同的子目录。

安装 openblas

修改环境变量，升级gcc为4.6或以上版本

[fangjin@caffe ~]$ jumbo install gcc46
[fangjin@caffe ~]$ export PATH=${JUMBO_ROOT}/bin:${JUMBO_ROOT}/opt/gcc46/bin:$PATH
[fangjin@caffe ~]$ source ~/.bashrc

下载安装openblas

[fangjin@caffe ~]$ git clone https://github.com/xianyi/OpenBLAS.git
[fangjin@caffe ~]$ cd OpenBLAS
[fangjin@caffe ~]$ make CC=gcc FC=gfortran
[fangjin@caffe ~]$ make PREFIX=/home/users/fangjin/opt install

其中/home/users/fangjin/opt要使用完整路径, 不要使用~/opt这种。

安装protobuf

jumbo 有现成的protobuf，可不必源码安装。此小节可略过。

[fangjin@caffe ~]$ git clone https://github.com/google/protobuf.git
[fangjin@caffe ~]$ cd protobuf-2.5.0
[fangjin@caffe ~]$ ./configure --prefix=/home/users/fangjin/opt
[fangjin@caffe ~]$ make
[fangjin@caffe ~]$ make install

安装boost

在网站下载最新源码包，例如boost_1_56_0.tar.gz

[fangjin@caffe ~]$ tar zxvf boost_1_56_0.tar.gz
[fangjin@caffe ~]$ cd boost_1_56_0
[fangjin@caffe ~]$ ./bootstrap.sh  --prefix=/home/users/fangjin/opt #通过执行 ./bootstrap.sh  --help来查看
[fangjin@caffe ~]$ ./b2 install #查看help也可在此步骤来指定--prefix

安装lmdb

[fangjin@caffe ~]$ git clone https://github.com/LMDB/lmdb.git
[fangjin@caffe ~]$ cd lmdb/libraries/liblmdb/
[fangjin@caffe ~]$ make
[fangjin@caffe ~]$ make prefix=/home/users/fangjin/opt install  #从src中Makefile中可以看到有一个prefix变量

安装python依赖库

如果需要编译python接口，需要安装相关依赖库。
内网环境无法连接http://pypi.douban.com/，可以参考pip.baidu.com.

#先安装pip
[fangjin@caffe ~]$ jumbo install python python-pip

将pypi.douban源改成baidu的源。

mkdir -p $HOME/.pip/
echo -e "[global]\nindex-url = http://pip.baidu.com/pypi/simple" >> $HOME/.pip/pip.conf
echo -e "[install]\ntrusted-host = pip.baidu.com" >> $HOME/.pip/pip.conf
echo -e "[easy_install]\nindex-url = http://pip.baidu.com/pypi/simple" >> $HOME/.pydistutils.cfg

注意，这种方法pip search无法使用。
安装相关依赖。

[fangjin@caffe ~]$ pip install --upgrade pip
[fangjin@caffe ~]$ cd caffe/python
[fangjin@caffe ~]$ for i in $(cat requirements.txt); do pip install "$i" ;done

配置安装项

将Makefile.config.example复制一份为Makefile.config

[fangjin@caffe ~]$ cp Makefile.config.example Makefile.config

修改Makefile.config中的配置
根据自己的需求来修改，注意的地方有两个
一个是PYTHON_INCLUDE 的路径，一个是PYTHON_LIB 的路径
然后自己添加LOCAL_PATH的路径。
INCLUDE_DIRS和LIBRARY_DIRS定义用到的库和头文件。

一个Makefile.config的示例如下

# Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!
# cuDNN acceleration switch (uncomment to build with cuDNN).
# USE_CUDNN := 1
# CPU-only switch (uncomment to build without GPU support).
CPU_ONLY := 1
# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++
# CUDA directory contains bin/ and lib/ directories that we need.
#CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr
# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.
#CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
                -gencode arch=compute_20,code=sm_21 \
                -gencode arch=compute_30,code=sm_30 \
                -gencode arch=compute_35,code=sm_35 \
                -gencode arch=compute_50,code=sm_50 \
                -gencode arch=compute_50,code=compute_50
# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := open
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas
# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib
# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app
# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /home/users/fangjin/.jumbo/include/python2.7 \
                /home/users/fangjin/.jumbo/lib/python2.7/site-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
                # $(ANACONDA_HOME)/include/python2.7 \
                # $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \
# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /home/users/zouxidong/.jumbo/lib/python2.7
# PYTHON_LIB := $(ANACONDA_HOME)/lib
# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib
# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include \
        ${JUMBO_ROOT}/include \
         ${JUMBO_ROOT}/opt/gcc46/include \
        /home/users/fangjin/opt/include 
LIBRARY_DIRS := $(PYTHON_LIB) \
        ${JUMBO_ROOT}/lib \
        ${JUMBO_ROOT}/opt/gcc46/lib64 \
       /home/users/fangjin/opt/lib
# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib
# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1
BUILD_DIR := build
DISTRIBUTE_DIR := distribute
# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1
# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0
# enable pretty build (comment to see full commands)
Q ?= @

同时，在.bashrc中需要定义之前安装的各种依赖库的bin和lib。

# export PATH=/home/users/fangjin/opt/bin:${JUMBO_ROOT}/bin:${JUMBO_ROOT}/opt/gcc46/bin:$PATH
# export LD_LIBRARY_PATH=/home/users/fangjin/opt/lib:${JUMBO_ROOT}/lib:${JUMBO_ROOT}/opt/gcc46/lib64:${LD_LIBRARY_PATH}

然后

[fangjin@caffe ~]$ source ~/.bashrc

安装

执行make -j8 编译caffe，运行成功时会生成libcaffe.so
执行make pycaffe输出如下内容PROTOC (python) src/caffe/proto/caffe.proto表示安装成功

测试

如何知道是否正确安装呢？
编译后的文件在caffe/build中。
可以运行build/tools/caffe看是否可以执行。
来利用提供的example做测试。

# 下载数据集
[fangjin@caffe ~]$ sh data/mnist/get_mnist.sh
# 转换数据集
[fangjin@caffe ~]$ sh examples/mnist/create_mnist.sh
# 开始训练
[fangjin@caffe ~]$ sh examples/mnist/train_lenet.sh

注意，如果不是使用GPU，可以看看train_lenet.sh中的solver文件，修改成cpu。

下面测试python的caffe接口是否安装成功。
可以将caffe接口直接写入PATH变量。
export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH
也可以直接将caffe/python/caffe复制到/home/users/fangjin/.jumbo/lib/python2.7/site-packages/
然后

[fangjin@caffe ~]$ source ~/.bashrc

1. 特别注意，路径是caffe/python，而不是caffe/python/caffe，否则numpy会报错。
2. export新增环境变量之后，即使修改了再次source ~/.bashrc，因为是使用:,前一次的变量值依旧保存，最简单的方法是退出终端，重新连接。

测试python接口

[fangjin@caffe ~]$ python
Python 2.7.3 (default, Aug  4 2016, 12:06:54) 
[GCC 3.4.5 20051201 (Red Hat 3.4.5-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>>

不报错，则成功。

但是开发机上安装基本都会报错

>>> import caffe
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/users/fangjin/my_workplace/c_workplace/010_learn_caffe/source_code/caffe/python/caffe/__init__.py", line 1, in <module>
    from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
  File "/home/users/fangjin/my_workplace/c_workplace/010_learn_caffe/source_code/caffe/python/caffe/pycaffe.py", line 15, in <module>
    import caffe.io
  File "/home/users/fangjin/my_workplace/c_workplace/010_learn_caffe/source_code/caffe/python/caffe/io.py", line 2, in <module>
    import skimage.io
  File "/home/users/fangjin/.jumbo/lib/python2.7/site-packages/skimage/io/__init__.py", line 7, in <module>
    from .manage_plugins import *
  File "/home/users/fangjin/.jumbo/lib/python2.7/site-packages/skimage/io/manage_plugins.py", line 28, in <module>
    from .collection import imread_collection_wrapper
  File "/home/users/fangjin/.jumbo/lib/python2.7/site-packages/skimage/io/collection.py", line 12, in <module>
    from PIL import Image
  File "/home/users/fangjin/.jumbo/lib/python2.7/site-packages/PIL/Image.py", line 67, in <module>
    from PIL import _imaging as core
ImportError: /home/users/fangjin/.jumbo/lib/python2.7/site-packages/PIL/_imaging.so: undefined symbol: TIFFSetWarningHandlerExt

单独from PIL import Image 也会出错，所以问题与caffe无关，具体原因经过分析，应该是libtiff库引起的。旧版本的libtiff库没有TIFFSetWarningHandlerExt，所以需要重新安装一个libtiff库。

下载地址http://dl.maptools.org/dl/libtiff/，下载tiff-3.8.2.tar.gz这个版本基本够用。原先的libtiff库文件安装在/usr/include和/usr/lib下，没有权限进行删除，这里安装到新路径下/home/fangjin/opt。

安装

[fangjin@com download]$ tar zxvf tiff-3.8.2.tar.gz
[fangjin@com download]$ cd tiff-3.8.2
[fangjin@com tiff-3.8.2]$ ./configure --prefix=/home/fangjin/opt/
[fangjin@com tiff-3.8.2]$ make

会报错

../libtool: eval: line 6176: unexpected EOF while looking for matching `''
../libtool: eval: line 6177: syntax error: unexpected end of file
make[2]: *** [libtiffxx.la] Error 1
make[2]: Leaving directory `/home/data/fangjin/download/tiff-3.8.2/libtiff'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/data/fangjin/download/tiff-3.8.2/libtiff'
make: *** [all-recursive] Error 1

按照提示修改libtool文件，将6176行中的的”$cmd”变量上双引号，换成单引号即可。
重新

[fangjin@com tiff-3.8.2]$ make; make install

把tiff库的路径加入到Makefile.config和PATH路径中(已经有则不需要重复添加)，重新生成caffe和pycaffe。
再尝试python接口，应该可以提示运行成功了。