Centos7 安装 Caffe 记录

最新推荐文章于 2023-07-04 10:36:25 发布

勾兑的果汁

最新推荐文章于 2023-07-04 10:36:25 发布

阅读量2.3k

点赞数 1

分类专栏：流程备忘文章标签：深度学习 Centos7 Caffe Docker

本文链接：https://blog.csdn.net/tuomen5867/article/details/94406903

版权

流程备忘专栏收录该内容

1 篇文章 0 订阅

订阅专栏

1. 背景

听闻 Google 为最近为 Dropout 申请专利，加上前段时间 Google 禁止华为手机使用 Android 系统的新闻被传得沸沸扬扬之后，总感觉 Google 会有更大的动作。目前 TensorFlow 已经成为最受欢迎且使用最广的深度学习框架（没有之一），被扼住脖子的感觉愈发强烈。股市中流传很广的一句话就是“不能把鸡蛋全部放在一个篮子里”，同样在深度学习领域中，想成为一名优秀的工程师，就不能局限住自己，不能只认一条路。
目前作为诸多任务 backbones 的优秀的卷积神经网络（如VggNet系列、ResNet系列等）中有很多是在 Caffe 框架下训练得到，而且 Caffe 作为一个比较成熟的深度学习框架，无论是在学界还是工业界都能满足人们的要求，所以趁着周末有时间来折腾下，顺便学习下相关的知识。闲话少叙，下面正式开始。

2. 基本环境

由于在服务器上进行操作，为了不对整体环境造成影响，所以在 docker 容器中进行安装。可以使用命令docker pull nvidia/cuda:8.0-cudnn6-devel-centos7从 DockerHub 中拉去镜像。镜像拉取成功之后可以通过docker images命令进行查看。这个被拉取的镜像本身已经配置好了 cuda 和 cudnn 环境，十分方便。下表是对所用 docker 容器中相关配置的详细介绍：

类别	版本	备注
OS	CentOS Linux release 7.6.1810 (Core)	切换了阿里的yum源和epel源
CUDA	8.0	无需配置
CUDNN	6.0	无需配置
Python	Anaconda3-4.2.0-Linux-x86_64.sh	Python3.5
GPU	Tesla M40 24GB	双显卡

安装必要软件：

yum install git
yum install vim
yum install make
yum install wget

3. 安装步骤

安装 Anaconda 的命令：sh Anaconda3-4.2.0-Linux-x86_64.sh，下边开始进行详细的配置过程。

更换 yum 源和 epel 源：

# 更换 yum 源
cp /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
# 更换 epel 源
wget -P /etc/yum.repos.d/ http://mirrors.aliyun.com/repo/epel-7.repo
# 清理缓存并生成新的缓存
yum clean all
yum makecache

更换 pip 源和 conda 源

# Linux 下修改 pip 源：
# 修改 ~/.pip/pip.conf (没有就创建一个)，添加如下内容：
[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple
# Linux 下修改 conda 源：
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --set show_channel_urls yes

安装 Caffe 需要的依赖库

yum install protobuf-devel leveldb-devel snappy-devel opencv-devel boost-devel hdf5-devel
yum install gflags-devel glog-devel lmdb-devel

将与 CUDA 和 CUDNN 相关路径设置成环境变量

# 将下面两行内容添加至文件 ~/.bashrc 最后：
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:/usr/local/cuda-8.0/extras/CUPTI/lib64:$LD_LIBRARY_PATH
export CUDA_HOME=/usr/local/cuda-8.0/
# 添加成功之后执行：
source ~/.bashrc

安装 BLAS（可选）

yum install atlas-devel
cd /usr/lib64/atlas
ln -sv libsatlas.so.3.10 libcblas.so
ln -sv libsatlas.so.3.10 libatlas.so

下载 Caffe 源码
从 https://github.com/BVLC/caffe.git 进行下载

git clone https://github.com/BVLC/caffe.git

安装 Python 依赖

# 进入 ${CAFFE_HOME}/python/，执行
for req in $(cat requirements.txt); do pip install $req; done

更改 Caffe 的配置文件

# 进入 Caffe 所在目录 CAFFE_HOME
cp Makefile.config.example Makefile.config # 做好备份
vim Makefile.conig # 打开文件并编辑

打开文件之后，按照如下内容酌情更改

## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0
# This code is taken from https://github.com/sh1r0/caffe-android-lib
# USE_HDF5 := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#	You should not set this flag if you will be reading LMDBs with any
#	possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3
# OPENCV_VERSION := 3

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda-8.0
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
		-gencode arch=compute_20,code=sm_21 \
		-gencode arch=compute_30,code=sm_30 \
		-gencode arch=compute_35,code=sm_35 \
		-gencode arch=compute_50,code=sm_50 \
		-gencode arch=compute_52,code=sm_52 \
		-gencode arch=compute_60,code=sm_60 \
		-gencode arch=compute_61,code=sm_61 \
		-gencode arch=compute_61,code=compute_61

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
BLAS_INCLUDE := /usr/include/atlas
BLAS_LIB := /usr/lib64/atlas

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# PYTHON_INCLUDE := /usr/include/python2.7 \
# 		/usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
ANACONDA_HOME := $(HOME)/anaconda3
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
		$(ANACONDA_HOME)/include/python3.6m \
		$(ANACONDA_HOME)/lib/python3.6/site-packages/numpy/core/include

# Uncomment to use Python 3 (default is Python 2)
PYTHON_LIBRARIES := boost_python python3.6m
# PYTHON_INCLUDE := /usr/include/python3.5m \
#                 /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
# PYTHON_LIB := /usr/lib
PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/local/cuda-8.0/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/local/cuda-8.0/lib

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @

编译 Caffe

make all
make test
make runtest

执行make runtest时可能会遇到错误：error while loading shared libraries: libhdf5_hl.so.10: cannot open shared object file: No such file or directory，这是由于版本不匹配而出现的问题，解决办法：

# 进入 /usr/lib64/
cd /usr/lib64/
# 查看相应的文件
ll libhdf5*
-rw-r--r-- 1 root root    4274 Aug 22  2017 libhdf5.settings
lrwxrwxrwx 1 root root      16 Jul  1 09:53 libhdf5.so -> libhdf5.so.8.0.1
lrwxrwxrwx 1 root root      16 Jul  1 09:51 libhdf5.so.8 -> libhdf5.so.8.0.1
-rwxr-xr-x 1 root root 4177552 Aug 22  2017 libhdf5.so.8.0.1
lrwxrwxrwx 1 root root      20 Jul  1 09:53 libhdf5_cpp.so -> libhdf5_cpp.so.8.0.1
lrwxrwxrwx 1 root root      20 Jul  1 09:51 libhdf5_cpp.so.8 -> libhdf5_cpp.so.8.0.1
-rwxr-xr-x 1 root root  347944 Aug 22  2017 libhdf5_cpp.so.8.0.1
lrwxrwxrwx 1 root root      24 Jul  1 09:53 libhdf5_fortran.so -> libhdf5_fortran.so.8.0.1
lrwxrwxrwx 1 root root      24 Jul  1 09:51 libhdf5_fortran.so.8 -> libhdf5_fortran.so.8.0.1
-rwxr-xr-x 1 root root  299176 Aug 22  2017 libhdf5_fortran.so.8.0.1
lrwxrwxrwx 1 root root      19 Jul  1 09:53 libhdf5_hl.so -> libhdf5_hl.so.8.0.1
lrwxrwxrwx 1 root root      19 Jul  1 09:51 libhdf5_hl.so.8 -> libhdf5_hl.so.8.0.1
-rwxr-xr-x 1 root root  144968 Aug 22  2017 libhdf5_hl.so.8.0.1
lrwxrwxrwx 1 root root      23 Jul  1 09:53 libhdf5_hl_cpp.so -> libhdf5_hl_cpp.so.8.0.1
lrwxrwxrwx 1 root root      23 Jul  1 09:51 libhdf5_hl_cpp.so.8 -> libhdf5_hl_cpp.so.8.0.1
-rwxr-xr-x 1 root root   11336 Aug 22  2017 libhdf5_hl_cpp.so.8.0.1
lrwxrwxrwx 1 root root      26 Jul  1 09:53 libhdf5hl_fortran.so -> libhdf5hl_fortran.so.8.0.1
lrwxrwxrwx 1 root root      26 Jul  1 09:51 libhdf5hl_fortran.so.8 -> libhdf5hl_fortran.so.8.0.1
-rwxr-xr-x 1 root root   99208 Aug 22  2017 libhdf5hl_fortran.so.8.0.1
# 执行
ln -sv libhdf5.so.8.0.1 libhdf5.so.10
ln -sv libhdf5_hl.so.8.0.1 libhdf5_hl.so.10

另外，还需进行一步操作：

# 编辑 ~/.vimrc
vim ~/.bashrc
# 把以下内容填入文件末尾，保存后执行 source ~/.bashrc
export HDF5_DISABLE_VERSION_CHECK=1

上述两步操作完成后执行 make runtest。结束之后， Caffe 便编译完成。

4. 测试 Caffe

运行 Caffe
进入Caffe 安装目录,执行./build/tools/caffe，可以根据caffe命令选项使用

# 看到如下信息说明安装成功
caffe: command line brew
usage: caffe <command> <args>

commands:
  train           train or finetune a model
  test            score a model
  device_query    show GPU diagnostic information
  time            benchmark model execution time

  Flags from tools/caffe.cpp:
    -gpu (Optional; run in GPU mode on given device IDs separated by ','.Use
      '-gpu all' to run on all available GPUs. The effective training batch
      size is multiplied by the number of devices.) type: string default: ""
    -iterations (The number of iterations to run.) type: int32 default: 50
    -level (Optional; network level.) type: int32 default: 0
    -model (The model definition protocol buffer text file.) type: string
      default: ""
    -phase (Optional; network phase (TRAIN or TEST). Only used for 'time'.)
      type: string default: ""
    -sighup_effect (Optional; action to take when a SIGHUP signal is received:
      snapshot, stop or none.) type: string default: "snapshot"
    -sigint_effect (Optional; action to take when a SIGINT signal is received:
      snapshot, stop or none.) type: string default: "stop"
    -snapshot (Optional; the snapshot solver state to resume training.)
      type: string default: ""
    -solver (The solver definition protocol buffer text file.) type: string
      default: ""
    -stage (Optional; network stages (not to be confused with phase), separated
      by ','.) type: string default: ""
    -weights (Optional; the pretrained weights to initialize finetuning,
      separated by ','. Cannot be set simultaneously with snapshot.)
      type: string default: ""

2.MNIST 例子
进入Caffe目录，执行如下命令

# 下载数据集：
./data/mnist/get_mnist.sh
# 转换数据集：
./examples/mnist/create_mnist.sh
# 训练例子：
./examples/mnist/train_lenet.sh

训练输出示例如下：

./examples/mnist/train_lenet.sh
I0628 15:12:45.510594 54419 caffe.cpp:218] Using GPUs 0
I0628 15:12:46.025655 54419 caffe.cpp:223] GPU 0: Tesla M60
I0628 15:12:46.311704 54419 solver.cpp:44] Initializing solver from parameters:
test_iter: 100
test_interval: 500
base_lr: 0.01
display: 100
max_iter: 10000
lr_policy: "inv"
gamma: 0.0001
power: 0.75
momentum: 0.9
weight_decay: 0.0005
snapshot: 5000
snapshot_prefix: "examples/mnist/lenet"
...
I0628 15:13:12.870817 54419 sgd_solver.cpp:273] Snapshotting solver state to binary proto file examples/mnist/lenet_iter_10000.solverstate
I0628 15:13:12.874179 54419 solver.cpp:310] Iteration 10000, loss = 0.00265072
I0628 15:13:12.874205 54419 solver.cpp:330] Iteration 10000, Testing net (#0)
I0628 15:13:12.962249 54426 data_layer.cpp:73] Restarting data prefetching from start.
I0628 15:13:12.964479 54419 solver.cpp:397]     Test net output #0: accuracy = 0.9906
I0628 15:13:12.964511 54419 solver.cpp:397]     Test net output #1: loss = 0.0283195 (* 1 = 0.0283195 loss)
I0628 15:13:12.964524 54419 solver.cpp:315] Optimization Done.
I0628 15:13:12.964529 54419 caffe.cpp:259] Optimization Done.

5. Caffe 的 Python 接口

执行一下命令即可得到 Caffe 的 Python 接口

# 进入 CAFFE_HOME
make pycaffe
# 在 ~/.bahsrc 添加以下内容，之后执行 source ~/.bashrc
export PATH=/root/caffe/python:$PATH

参考

勾兑的果汁

关注

1
点赞
踩
10

收藏

觉得还不错? 一键收藏
3
评论
Centos7 安装 Caffe 记录

1. 背景听闻 Google 为最近为 Dropout 申请专利，加上前段时间 Google 禁止华为手机使用 Android 系统的新闻被传得沸沸扬扬之后，总感觉 Google 会有更大的动作。目前 TensorFlow 已经成为最受欢迎且使用最广的深度学习框架（没有之一），被扼住脖子的感觉愈发强烈。股市中流传很广的一句话就是“不能把鸡蛋全部放在一个篮子里”，同样在深度学习领域中，想成为一名优...
复制链接

扫一扫