ubuntu16.04+CUDA9.0+Anaconda3.0环境下Caffe安装过程及问题

最新推荐文章于 2024-02-27 18:17:12 发布

实验室的搬运工

最新推荐文章于 2024-02-27 18:17:12 发布

阅读量684

点赞数 1

分类专栏： Ubuntu

本文链接：https://blog.csdn.net/qq_34570910/article/details/99410167

版权

Ubuntu 专栏收录该内容

52 篇文章 1 订阅

订阅专栏

1. 安装caffe for python的虚拟环境

conda create -n caffe_py27 pip python=2.7

2.安装第三方库和工具

sudo apt-get update
sudo apt-get install -y build-essential cmake git pkg-config
sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install -y libatlas-base-dev
sudo apt-get install -y libboost-all-dev
sudo apt-get install -y libgflags-dev libgoogle-glog-dev liblmdb-dev
sudo apt-get install -y python-pip
sudo apt-get install -y python-dev
sudo apt-get install -y python-numpy python-scipy

sudo apt install cmake
sudo apt install unzip
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
sudo apt-get install libleveldb-dev
sudo apt-get install libsnappy-dev
sudo apt-get install libatlas-base-dev
sudo apt-get install libhdf5-serial-dev
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt install protobuf-compiler
sudo apt-get install libprotobuf-dev
sudo apt install python-pip

sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler

对于没有安装opencv的读者可以执行下一行命令：

sudo apt-get install libopencv-dev

3.下载caffe，进入caffe目录中ssd分支：

git clone https://github.com/weiliu89/caffe.git
cd caffe
git checkout ssd

cd caffe/python/

for req in $(cat requirements.txt); do sudo pip install $req; done

4.修改caffe配置文件Makefile.config

cp Makefile.config.example Makefile.config

gedit Makefile.config

取消

#USE_CUDNN:= 1

#OPENCV_VERSION:= 3

#WITH_PYTHON_LAYER:= 1

前的#注释符

PYTHON_HOME变量配置。因为使用的是Python是Anaconda配置的虚拟环境，因此需要将python的环境指向该地址。如图二所示，我的Anaconda安装在$(HOME)/anaconda3位置，虚拟环境创建以后会在envs下创建一个与虚拟环境同名的文件夹，我直接将anaconda的目录指向该位置。PYTHON的路径基于该位置指定。配置结果如下图所示。

如果cuda 的版本号>=9.0，则在该文件夹下找到CUDA_ARCH变量，并把sm_20和sm_21两行去掉。下面是最后保留部分

CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \
       -gencode arch=compute_35,code=sm_35 \
       -gencode arch=compute_50,code=sm_50 \
       -gencode arch=compute_52,code=sm_52 \
       -gencode arch=compute_60,code=sm_60 \
       -gencode arch=compute_61,code=sm_61 \
       -gencode arch=compute_61,code=compute_61

修改PYTHON_LIB、INCLUDE_DIRS和LIBRARY_DIRS变量配置

5.修改caffe配置文件Makefile

将：
NVCCFLAGS +=-ccbin=$(CXX) -Xcompiler-fPIC $(COMMON_FLAGS)
替换为：
NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)

将：
LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_hl hdf5
改为：
LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl hdf5_serial

6.然后修改 /usr/local/cuda/include/host_config.h 文件 :

将
#error-- unsupported GNU version! gcc versions later than 4.9 are not supported!
改为
//#error-- unsupported GNU version! gcc versions later than 4.9 are not supported!

7.编译

make all -j8

问题及解决方案

/usr/include/c++/5/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
 #error This file requires compiler and library support \

可以在Makefile文件中添加-std=c++11解决

CXXFLAGS += -pthread -fPIC $(COMMON_FLAGS) $(WARNINGS) -std=c++11

/usr/include/c++/5/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support must be enabled with the -std=c++11 or -std=gnu++11 compiler options.

需要改makefile文件：在对应的行后加上-std=c++11
NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS) -std=c++11
LINKFLAGS += -pthread -fPIC $(COMMON_FLAGS) $(WARNINGS) -std=c++11

参考:http://www.manongjc.com/article/53635.html

/usr/bin/ld: cannot find -lopencv_imgcodecs

为了改正错误尝试修改Makefile.config文件,加入了opencv的路径：

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial /usr/local/opencv3.4.6/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial /usr/local/opencv3.4.6/lib

CXX/LD -o .build_release/tools/extract_features.bin
//home/wds/anaconda3/lib/libpng16.so.16：对‘inflateValidate@ZLIB_1.2.9’未定义的引用

在caffe目录下修改Makefile.config，添加Anaconda 路径：

export LD_LIBRARY_PATH=/home/ubuntu/anaconda3/lib:$LD_LIBRARY_PATH

/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFReadRGBAStrip@LIBTIFF_4.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFIsTiled@LIBTIFF_4.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFWriteScanline@LIBTIFF_4.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFGetField@LIBTIFF_4.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFScanlineSize@LIBTIFF_4.0’未定义的引用
//usr/lib/x86_64-linux-gnu/libSM.so.6：对‘uuid_generate@UUID_1.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFReadEncodedTile@LIBTIFF_4.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFReadRGBATile@LIBTIFF_4.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFClose@LIBTIFF_4.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFRGBAImageOK@LIBTIFF_4.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFOpen@LIBTIFF_4.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFReadEncodedStrip@LIBTIFF_4.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFSetField@LIBTIFF_4.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFSetWarningHandler@LIBTIFF_4.0’未定义的引用
//usr/lib/x86_64-linux-gnu/libSM.so.6：对‘uuid_unparse_lower@UUID_1.0’未定义的引用
/usr/lib/x86_64-linux-gnu/libopencv_highgui.so：对‘TIFFSetErrorHandler@LIBTIFF_4.0’未定义的引用

这个问题是系统存在多个opencv库导致的,我们用ldd命令列出动态库依赖关系：

发现是opencv版本指向了2.4×××，它在LD_LIBRARY_PATH的优先级更高，与系统安装的opencv3.4.6冲突，应该是之前执行sudo apt-get install libopencv-dev时安装了opencv2.4.9，而这个版本仅依赖，因此先将它卸载。

sudo apt-get autoremove libopencv-dev

卸载干净时会出现下面情况：

再次编译就成功了。

.build_release/lib/libcaffe.so：对‘boost::re_detail::cpp_regex_traits_implementation<char>::transform_primary(char const*, char const*) const’未定义的引用
.build_release/lib/libcaffe.so：对‘boost::re_detail::cpp_regex_traits_implementation<char>::transform(char const*, char const*) const’未定义的引用
.build_release/lib/libcaffe.so：对‘boost::match_results<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >::maybe_assign(boost::match_results<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<boost::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > const&)’未定义的引用

1.)You could add boost_regex at the end of LIBARTIES in Makefile.(line 181)

2.)重新从源码上构建boost.

//usr/lib/x86_64-linux-gnu/libSM.so.6：对‘uuid_generate@UUID_1.0’未定义的引用
//usr/lib/x86_64-linux-gnu/libSM.so.6：对‘uuid_unparse_lower@UUID_1.0’未定义的引用

分析：

ldd /usr/lib/x86_64-linux-gnu/libSM.so 查看依赖库发现libuuid指向anaconda3库,而系统的库却在另一个地方。

ldconfig -p|grep uuid
	libuuid.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libuuid.so.1

而undefined reference to 在寻找链接库时有个原则，如果搜索路径里存在同名的目标，以给定顺序先后确定优先级，只要查找到第一个就会返回,后面的不会再查找。

把anaconda2/lib下的libuuid库文件移走就好了

ls ~/anaconda2/lib/libuuid*
 
mkdir ~/anaconda2/libuuid
 
mv ~/anaconda2/lib/libuuid* ~/anaconda2/libuuid

make all 成功后执行make test -j32

8. make test -j4

报错：

/usr/include/c++/5/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and

缺少 -std=c++11, 在Makefile文件中以下内容后面加入 -std=c++11.

CXXFLAGS += -pthread -fPIC $(COMMON_FLAGS) $(WARNINGS)
NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)
LINKFLAGS += -pthread -fPIC $(COMMON_FLAGS) $(WARNINGS)

再次编译即可成功。

编译成功后可运行测试：

make runtest

.build_release/tools/caffe: error while loading shared libraries: libcudart.so.9.0: cannot open shared object file: No such file or directory
Makefile:537: recipe for target 'runtest' failed

解决方法：

1.）执行sudo ldconfig /usr/local/cuda/lib64 （生效一次）

2.）将/usr/local/cuda/lib64 添加至/etc/ld.so.conf.d/cuda.conf文件执行 sudo ldconfig （永久生效）

10.继续执行make runtest

.build_release/tools/caffe
.build_release/tools/caffe: error while loading shared libraries: libopencv_core.so.3.4: cannot open shared object file: No such file or directory
Makefile:537: recipe for target 'runtest' failed
make: *** [runtest] Error 127

添加opencv编译环境到系统，首先创建并编译.conf文件.

sudo gedit /etc/ld.so.conf.d/opencv.conf

添加 /usr/local/opencv3.4.6/lib 然后保存关闭

执行 sudo ldconfig 使环境配置生效。

11. 继续执行 sudo make runtest -j16

显示以上内容表示caffe安装成功。

12.安装pycaffe

cd caffe-master/python
source activate caffe_py35 #如果用anaconda环境
for req in $(cat requirements.txt); do pip install $req; done
cd ..
make pycaffe -j8

12.1在python中执行import caffe报错，没有这个模块：

sudo gedit ~/.bashrc
在文档最后加上：
export PYTHONPATH=/home/wds/tool/Caffe/Caffe-ssd/caffe/python:$PYTHONPATH

12.2 在python中执行import caffe出错

ImportError: /usr/local/lib/libcudart.so.9.0: version `libcublas.so.9.0' not found (required by /home/wds/tool/Caffe/Caffe-ssd/caffe/python/caffe/../../build/lib/libcaffe.so.1.0.0-rc3)

然后，我查看了libcaffe的依赖：

ldd libcaffe.so.1.0.0-rc3 
./libcaffe.so.1.0.0-rc3: /usr/local/lib/libcudart.so.9.0: version `libcublas.so.9.0' not found (required by ./libcaffe.so.1.0.0-rc3)
./libcaffe.so.1.0.0-rc3: /lib/x86_64-linux-gnu/libz.so.1: version `ZLIB_1.2.9' not found (required by /home/wds/anaconda3/lib/libpng16.so.16)

我们先看zlib：

wds@wds:/usr/local/cuda/lib64$ find /usr/ -name zlib.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/zlib.pc
wds@wds:/usr/local/cuda/lib64$ cat /usr/lib/x86_64-linux-gnu/pkgconfig/zlib.pc 
prefix=/usr
exec_prefix=${prefix}
libdir=${prefix}/lib/x86_64-linux-gnu
sharedlibdir=${libdir}
includedir=${prefix}/include

Name: zlib
Description: zlib compression library
Version: 1.2.8

Requires:
Libs: -L${libdir} -L${sharedlibdir} -lz
Cflags: -I${includedir}

可以看到系统中zlib版本是1.2.8，所以我们升级到1.2.9，先下载，然后配置：

tar -xvf ~/Downloads/zlib-1.2.9.tar.gz
cd zlib-1.2.9
sudo -s
./configure; make; make install
cd /lib/x86_64-linux-gnu
ln -s -f /usr/local/lib/libz.so.1.2.9/lib libz.so.1
cd ~
rm -rf zlib-1.2.9

然后我们解决另外一个报错，重新建立软链接：

wds@wds:/usr/local/cuda/lib64$ sudo ln -s -f libcublas.so.9.0 /usr/local/lib/libcublas.so.9.0
wds@wds:/usr/local/cuda/lib64$ sudo ln -s -f libcudart.so.9.0 /usr/local/lib/libcudart.so.9.0

13.在另一台机器配置环境时，make all -j32,出错。

In file included from .build_release/src/caffe/proto/caffe.pb.cc:4:0:
.build_release/src/caffe/proto/caffe.pb.h:12:2: error: #error This file was generated by a newer version of protoc which is
 #error This file was generated by a newer version of protoc which is

原因： it is because of the wrong version of protobuf
首先你要卸载，并重装其要求的protobuf版本（2.6.1），之后重新编译就好了

卸载办法：

    #remove
    sudo apt-get remove libprotobuf-dev protobuf-compiler
    sudo apt-get remove libprotobuf-lite8 libprotoc8
    sudo apt-get remove python-protobuf
    sudo pip uninstall protobuf
    #如果安装了anaconda
    conda uninstall protobuf

如果出现错误：

E:Encountered a section with no Package: header,

输入以下命令：

sudo rm /var/lib/apt/lists/* -vf
sudo apt-get update

重新安装

    #reinstall
    sudo apt-get install libprotobuf-dev protobuf-compiler
    #if anaconda
    conda install -c anaconda protobuf=2.6.1
    或者
    /home/username/anaconda2/bin/pip install protobuf == 2.6.1

附注：

查看当前protoc版本：protoc --version

查看protoc安装位置：which protoc

查找protoc相关文件：sudo find / -name protoc

手动安装protobuf 2.6.1方法：

http://www.cnblogs.com/javaee6/p/4849051.html

14.

In file included from .build_release/src/caffe/proto/caffe.pb.cc:12:0:
/usr/local/include/google/protobuf/wire_format_lite_inl.h:42:43: fatal error: google/protobuf/stubs/logging.h: No such file or directory
compilation terminated.

Google上找到原因，是依赖没有装好！遂记录下来，免得大家百度的时候，搜到的只有上面两种答案，费时费力。

解决方案

在编译caffe前先安装一下库。

sudo apt-get install git
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev
libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libatlas-base-dev
sudo apt-get install python-dev
sudo apt-get insall libgflags-dev libgoogle-glog-dev liblmdb-dev

实验室的搬运工

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
ubuntu16.04+CUDA9.0+Anaconda3.0环境下Caffe安装过程及问题

1. 安装caffe for python的虚拟环境conda create -n caffe_py27 pip python=2.72.安装第三方库和工具sudo apt-get updatesudo apt-get install -y build-essential cmake git pkg-configsudo apt-get install -y libprotob...
复制链接

扫一扫