『DL笔记』Ubuntu16.04安装cuda-8.0+cudnn5.1+opencv-3.1+caffe+本地浏览器访问服务器jupyter notebook设置

最新推荐文章于 2021-10-15 11:09:14 发布
最新推荐文章于 2021-10-15 11:09:14 发布
阅读量725 收藏
点赞数 1
分类专栏： Machine Learning学习笔记 Deep Learning学习笔记 文章标签： Machine learning Deep learning caffee opencv cuda
引用请附上作者博客链接https://zhangkaifang.blog.csdn.net/，谢谢理解！
本文链接：https://blog.csdn.net/abc13526222160/article/details/102306967
版权
Ubuntu16.04安装cuda-8.0+cudnn+opencv-3.1+caffe教程！

文章目录

一、安转cuda+cudnn

cuda8.0+cudnn5.1百度云下载链接： 提取码：0ls9

1.1、安转cuda

  • 安装ssh工具：备注：这一步需要到服务器桌面上的命令窗口输入，这一步完成后，就可以用ssh工具远程连接服务器了，本文使用的是XShell。
sudo apt-get install openssh-server
  • 参考我之前的链接，方法一样的；安装cuda!

1.2、安转cudnn

二、安转opencv-3.1

2.1、安装依赖库

sudo apt-get install build-essential
 
sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
 
sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev
 
sudo apt-get install --assume-yes libopencv-dev libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff5-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libxine2-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libv4l-dev libtbb-dev libqt4-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev x264 v4l-utils unzip
 
sudo apt-get install ffmpeg libopencv-dev libgtk-3-dev python-numpy python3-numpy libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff5-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libxine2-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libv4l-dev libtbb-dev qtbase5-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev
  • 下载之后，解压到你要安装的位置，命令行进入已解压的文件夹 opencv-3.1.0 目录下，执行：
mkdir build # 创建编译的文件目录

cd build

cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..
  • 成功的话显示如下：
make -j8  #编译
make install

2.2、问题1: 下载ippicv_linux_20151201失败

  • 问题1：OpenCV 3.1下载 ippicv_linux_20151201失败;安装OpenCv 3.1的过程中要下载ippicv_linux_20151201，由于网络的原因，这个文件经常会下载失败。解决的办法是手动下载： ippicv_linux_20151201；将下载后的ippicv文件拷到下面这个目录，问题得到解决。

参考这个文章：opencv 解决ippicv下载慢的问题: ippicv_2019_lnx_intel64_general_20180723

opencv-3.1.0/3rdparty/ippicv/downloads/linux-808b791a6eac9ed78d32a7666804320e/
  • 最终cmake执行成功显示如下（部分）：
--   Python (for build):            /usr/bin/python2.7
-- 
--   Java:
--     ant:                         NO
--     JNI:                         NO
--     Java wrappers:               NO
--     Java tests:                  NO
-- 
--   Matlab:                        Matlab not found or implicitly disabled
-- 
--   Documentation:
--     Doxygen:                     NO
--     PlantUML:                    NO
-- 
--   Tests and samples:
--     Tests:                       YES
--     Performance tests:           YES
--     C/C++ Examples:              NO
-- 
--   Install path:                  /usr/local
-- 
--   cvconfig.h is in:              /home/zhangkf/opencv-3.1.0/build
-- -----------------------------------------------------------------
-- 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/zhangkf/opencv-3.1.0/build

2.3、问题2: CUDA 8.0不支持OpenCV的GraphCut算法

  • 问题2：由于CUDA 8.0不支持OpenCV的 GraphCut 算法，可能出现以下错误：
/home/zhangkf/opencv-3.1.0/modules/cudalegacy/src/graphcuts.cpp:275:146: error: ‘nppiGraphcut8_32f8u’ was not declared in this scope
             static_cast<int>(terminals.step), static_cast<int>(leftTransp.step), sznpp, labels.ptr<Npp8u>(), static_cast<int>(labels.step), state) );
                                                                                                                                                  ^
/home/zhangkf/opencv-3.1.0/modules/core/include/opencv2/core/private.cuda.hpp:165:52: note: in definition of macro ‘nppSafeCall’
 #define nppSafeCall(expr)  cv::cuda::checkNppError(expr, __FILE__, __LINE__, CV_Func)
                                                    ^
make[2]: *** [modules/cudalegacy/CMakeFiles/opencv_cudalegacy.dir/src/graphcuts.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [modules/cudalegacy/CMakeFiles/opencv_cudalegacy.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 90%] Linking CXX shared library ../../lib/libopencv_photo.so
[ 90%] Built target opencv_photo
make: *** [all] Error 2
  • 解决方法：修改 /opencv-3.1.0/modules/cudalegacy/src/graphcuts.cpp 文件内容，如下：
#include "precomp.hpp"
#if !defined (HAVE_CUDA) || defined (CUDA_DISABLER)

改为

#include "precomp.hpp"
#if !defined (HAVE_CUDA) || defined (CUDA_DISABLER) || (CUDART_VERSION >= 8000)
  • 最终make -j8 #编译成功显示如下（部分）：
[100%] Building CXX object modules/superres/CMakeFiles/opencv_test_superres.dir/test/test_main.cpp.o
[100%] Building CXX object modules/superres/CMakeFiles/opencv_test_superres.dir/test/test_superres.cpp.o
[100%] Building CXX object modules/superres/CMakeFiles/opencv_perf_superres.dir/perf/perf_superres.cpp.o
[100%] Building CXX object modules/superres/CMakeFiles/opencv_perf_superres.dir/perf/perf_main.cpp.o
[100%] Linking CXX executable ../../bin/opencv_test_superres
[100%] Built target opencv_test_superres
[100%] Linking CXX executable ../../bin/opencv_perf_superres
[100%] Built target opencv_perf_superres
Scanning dependencies of target opencv_videostab
[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/frame_source.cpp.o
[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/global_motion.cpp.o
[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/stabilizer.cpp.o
[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/inpainting.cpp.o
[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/fast_marching.cpp.o
[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/deblurring.cpp.o
[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/wobble_suppression.cpp.o
[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/outlier_rejection.cpp.o
[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/motion_stabilizing.cpp.o
[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/log.cpp.o
[100%] Building CXX object modules/videostab/CMakeFiles/opencv_videostab.dir/src/optical_flow.cpp.o
[100%] Linking CXX shared library ../../lib/libopencv_videostab.so
[100%] Built target opencv_videostab
[100%] Generating pyopencv_generated_include.h, pyopencv_generated_funcs.h, pyopencv_generated_types.h, pyopencv_generated_type_reg.h, pyopencv_generated_ns_reg.h
[100%] Generating pyopencv_generated_include.h, pyopencv_generated_funcs.h, pyopencv_generated_types.h, pyopencv_generated_type_reg.h, pyopencv_generated_ns_reg.h
Scanning dependencies of target opencv_python2
Scanning dependencies of target opencv_python3
[100%] Building CXX object modules/python2/CMakeFiles/opencv_python2.dir/__/src2/cv2.cpp.o
[100%] Building CXX object modules/python3/CMakeFiles/opencv_python3.dir/__/src2/cv2.cpp.o
[100%] Linking CXX shared module ../../lib/python3/cv2.cpython-34m.so
[100%] Built target opencv_python3
[100%] Linking CXX shared module ../../lib/cv2.so
[100%] Built target opencv_python2
zhangkf@Ubuntu:~/opencv-3.1.0/build$
  • 安装完成后通过查看opencv版本验证是否安装成功：
pkg-config --modversion opencv

三、安转caffe(convolutional architecture for feature embedding)

3.1、caffe简介
  • 终于可以安装caffe了，使用Git直接下载Caffe非常简单，或者去https://github.com/BVLC/caffe下载。由于我习惯去github上找代码，所以就直接去下载的源码。
git clone https://github.com/BVLC/caffe.git
  • 进入 caffe ，将 Makefile.config.example 文件复制一份并更名为 Makefile.config ，也可以在 caffe 目录下直接调用以下命令完成复制操作 ：
sudo cp Makefile.config.example Makefile.config
  • 复制一份的原因是编译 caffe 时需要的是 Makefile.config 文件，而Makefile.config.example 只是caffe 给出的配置文件例子，不能用来编译 caffe。然后修改 Makefile.config 文件，在 caffe 目录下打开该文件：
sudo gedit Makefile.config

3.2、修改Makefile.config文件内容

  • 这里把我电脑上用到的具体配置贴出来：
## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1 ##########################

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
USE_OPENCV := 1 ##########################
# USE_LEVELDB := 0
# USE_LMDB := 0
# This code is taken from https://github.com/sh1r0/caffe-android-lib
# USE_HDF5 := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#	You should not set this flag if you will be reading LMDBs with any
#	possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3
OPENCV_VERSION :=3 ##########################

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.


CUDA_DIR := /usr/local/cuda-10.0 ##########################
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
# CUDA_ARCH := -gencode arch=compute_20,code=sm_20
#################################################################
CUDA_ARCH :=-gencode arch=compute_30,code=sm_30 \
		-gencode arch=compute_35,code=sm_35 \
		-gencode arch=compute_50,code=sm_50 \
		-gencode arch=compute_52,code=sm_52 \
		-gencode arch=compute_60,code=sm_60 \
		-gencode arch=compute_61,code=sm_61 \
		-gencode arch=compute_61,code=compute_61

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas ##########################
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
#BLAS_INCLUDE := /home/asrim/intel/mkl/include
#BLAS_LIB := /home/asrim/intel/mkl/lib/intel64

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
MATLAB_DIR := /usr/local ##########################MATLAB根据自己而定
MATLAB_DIR := /usr/local/MATLAB/R2016b ##########################MATLAB根据自己而定

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# PYTHON_INCLUDE := /usr/include/python2.7 \
#		/usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
ANACONDA_HOME := $(HOME)/anaconda3/envs/caffe ############################
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
		$(ANACONDA_HOME)/include/python3.5m \
		$(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include

# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python-py35 python3.5m
# PYTHON_INCLUDE := /usr/include/python3.5m \
#                 /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
# PYTHON_LIB := /usr/lib
PYTHON_LIB := $(ANACONDA_HOME)/lib #########################
LINKFLAGS := -Wl,-rpath,$(ANACONDA_HOME)/lib ####################

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial ##########
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial ##########

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @

注意一点： 我的依赖库都是在anaconda虚拟环境caffe中安装的，这里ANACONDA_HOME := $(HOME)/anaconda3/envs/caffe

  • 1、若使用cudnn，则将
#USE_CUDNN := 1 
修改为：
USE_CUDNN := 1
  • 2、若使用的opencv版本是3的，则将
#OPENCV_VERSION := 3 
修改为：
OPENCV_VERSION := 3
  • 3、若要使用python来编写layer，则将
#WITH_PYTHON_LAYER := 1 
修改为：
WITH_PYTHON_LAYER := 1
  • 4、修改 python 路径(重要的一项)
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib 
修改为： 
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial

3.3、修改caffe目录下的Makefile文件内容

将：
NVCCFLAGS +=-ccbin=$(CXX) -Xcompiler-fPIC $(COMMON_FLAGS)
替换为：
NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)

将：
LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_hl hdf5
改为：
LIBRARIES += glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl hdf5_serial
  • 然后修改 /usr/local/cuda/include/host_config.h 文件 :
将
#error-- unsupported GNU version! gcc versions later than 4.9 are not supported!
改为
//#error-- unsupported GNU version! gcc versions later than 4.9 are not supported!

3.4、编译

  • 开始编译，在 caffe 目录下执行 ：
make all -j8
  • 如果显示结果如下（部分），则表示 caffe 已经成功安装。
CXX src/caffe/layers/cudnn_tanh_layer.cpp
CXX src/caffe/layers/power_layer.cpp
CXX src/caffe/layers/input_layer.cpp
CXX src/caffe/layers/log_layer.cpp
CXX src/caffe/layers/conv_layer.cpp
CXX src/caffe/layers/lrn_layer.cpp
CXX src/caffe/layers/loss_layer.cpp
CXX src/caffe/layers/slice_layer.cpp
CXX src/caffe/layers/batch_norm_layer.cpp
CXX src/caffe/layers/cudnn_sigmoid_layer.cpp
CXX src/caffe/layers/tile_layer.cpp
CXX src/caffe/layers/mvn_layer.cpp
CXX src/caffe/layers/spp_layer.cpp
CXX src/caffe/layers/prelu_layer.cpp
CXX src/caffe/layers/accuracy_layer.cpp
CXX src/caffe/layers/clip_layer.cpp
CXX src/caffe/layers/threshold_layer.cpp
CXX src/caffe/layers/softmax_loss_layer.cpp
CXX src/caffe/layers/embed_layer.cpp
CXX src/caffe/layers/sigmoid_cross_entropy_loss_layer.cpp
CXX src/caffe/layers/tanh_layer.cpp
CXX src/caffe/layers/cudnn_lrn_layer.cpp
CXX src/caffe/layers/elu_layer.cpp
CXX src/caffe/layers/scale_layer.cpp
CXX src/caffe/layers/bnll_layer.cpp
CXX src/caffe/layers/lstm_unit_layer.cpp
CXX src/caffe/layers/image_data_layer.cpp
CXX src/caffe/layers/dropout_layer.cpp
CXX src/caffe/layers/filter_layer.cpp
CXX src/caffe/layers/swish_layer.cpp
CXX src/caffe/layers/neuron_layer.cpp
CXX src/caffe/layers/cudnn_conv_layer.cpp
CXX src/caffe/layers/hdf5_data_layer.cpp
CXX src/caffe/layers/reduction_layer.cpp
CXX src/caffe/layers/data_layer.cpp
CXX src/caffe/layers/split_layer.cpp
CXX src/caffe/layers/batch_reindex_layer.cpp
CXX src/caffe/layers/base_data_layer.cpp
CXX src/caffe/layers/relu_layer.cpp
CXX src/caffe/layers/base_conv_layer.cpp
CXX src/caffe/layers/bias_layer.cpp
CXX src/caffe/layers/parameter_layer.cpp
CXX src/caffe/layers/euclidean_loss_layer.cpp
CXX src/caffe/layers/cudnn_pooling_layer.cpp
CXX src/caffe/layers/dummy_data_layer.cpp
CXX src/caffe/layers/recurrent_layer.cpp
CXX src/caffe/layers/infogain_loss_layer.cpp
CXX src/caffe/layers/multinomial_logistic_loss_layer.cpp
CXX src/caffe/layers/sigmoid_layer.cpp
CXX src/caffe/layers/cudnn_softmax_layer.cpp
CXX src/caffe/layers/softmax_layer.cpp
CXX src/caffe/layers/rnn_layer.cpp
CXX src/caffe/layers/flatten_layer.cpp
CXX src/caffe/layers/contrastive_loss_layer.cpp
CXX src/caffe/layers/inner_product_layer.cpp
CXX src/caffe/layers/hinge_loss_layer.cpp
CXX src/caffe/layers/exp_layer.cpp
CXX src/caffe/layers/crop_layer.cpp
CXX src/caffe/layers/argmax_layer.cpp
CXX src/caffe/layers/absval_layer.cpp
CXX src/caffe/layers/cudnn_lcn_layer.cpp
CXX src/caffe/layers/cudnn_deconv_layer.cpp
CXX src/caffe/layers/lstm_layer.cpp
CXX src/caffe/layers/concat_layer.cpp
CXX src/caffe/layers/deconv_layer.cpp
CXX src/caffe/layers/hdf5_output_layer.cpp
CXX src/caffe/layers/im2col_layer.cpp
CXX src/caffe/layers/eltwise_layer.cpp
CXX src/caffe/layers/silence_layer.cpp
CXX src/caffe/layers/memory_data_layer.cpp
CXX src/caffe/layers/window_data_layer.cpp
CXX src/caffe/layers/pooling_layer.cpp
CXX src/caffe/layers/reshape_layer.cpp
CXX src/caffe/layers/cudnn_relu_layer.cpp
CXX src/caffe/util/db_lmdb.cpp
CXX src/caffe/util/math_functions.cpp
CXX src/caffe/util/benchmark.cpp
CXX src/caffe/util/db_leveldb.cpp
CXX src/caffe/util/hdf5.cpp
CXX src/caffe/util/im2col.cpp
CXX src/caffe/util/blocking_queue.cpp
CXX src/caffe/util/io.cpp
CXX src/caffe/util/insert_splits.cpp
CXX src/caffe/util/upgrade_proto.cpp
CXX src/caffe/util/db.cpp
CXX src/caffe/util/cudnn.cpp
CXX src/caffe/util/signal_handler.cpp
CXX src/caffe/parallel.cpp
CXX src/caffe/net.cpp
AR -o .build_release/lib/libcaffe.a
LD -o .build_release/lib/libcaffe.so.1.0.0
CXX/LD -o .build_release/tools/extract_features.bin
CXX/LD -o .build_release/tools/caffe.bin
CXX/LD -o .build_release/tools/convert_imageset.bin
CXX/LD -o .build_release/tools/upgrade_solver_proto_text.bin
CXX/LD -o .build_release/tools/upgrade_net_proto_binary.bin
CXX/LD -o .build_release/tools/upgrade_net_proto_text.bin
CXX/LD -o .build_release/tools/compute_image_mean.bin
CXX/LD -o .build_release/examples/mnist/convert_mnist_data.bin
CXX/LD -o .build_release/examples/cpp_classification/classification.bin
CXX/LD -o .build_release/examples/siamese/convert_mnist_siamese_data.bin
CXX/LD -o .build_release/examples/cifar10/convert_cifar_data.bin
zhangkf@wang-ThinkStation-P520:~/caffe$ clear

sudo make test -j8
  • 如果之前的配置或安装出错，那么编译就会出现各种各样的问题，所以前面的步骤一定要细心。编译成功后可运行测试：
sudo make runtest -j8
  • 然后，配置环境变量，将/home/zhangkf/anaconda3/envs/caffe/python加到环境变量:
export PYTHONPATH="/home/zhangkf/anaconda3/envs/caffe/bin/python:$PATH"
  • 然后输入Python
(caffe) zhangkf@wang-ThinkStation-P520:~$ python
Python 3.4.5 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:47:47) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>>
  • 上面如果 ImportError: No module named caffe，需要把caffe下的Python路径导入环境变量中去；
sudo vim ~/.bashrc
# 最后一行加上，这里的路径写上你自己的路径。
export PYTHONPATH="home/zhangkf/caffe/python:$PATH"
# 记得执行下面的命令，否则的话只能在这个目录下执行Python，导入caffe了；
source ~/.bashrc
  • 下面测试caffe是不是安装好

在这里插入图片描述

  • 想要不再特定目录下正常运行caffe这个命令，需要添加环境变量。这样任何位置输入caffe都可以啦。
alias caffe="/home/zhangkf/caffe/build/tools/caffe"

四、caffe简单例子

4.1、准备数据

  • mnist数据集下载，执行/home/zhangkf/caffe/data/mnist目录下的get_mnist.sh脚本进行mnist数据集下载
bash get_mnist.sh
  • 解压数据集
(caffe) zhangkf@wang-ThinkStation-P520:~/caffe/data/mnist$ tar -xvzf mnist.tar.gz 
t10k-images-idx3-ubyte
train-images-idx3-ubyte
t10k-labels-idx1-ubyte
train-labels-idx1-ubyte

在这里插入图片描述

  • 数据集格式转换；下载得到的MNIST数据集是二进制文件,需要转换为LEVELDB或者LMDB才能被Caffe识别. 可以通过Caffe框架的 /caffe/example/mnsit/create_mnist.sh 将原始数据转换为lmdb/leveldb 格式（注意：需要在caffe路径下运行）
(caffe) zhangkf@wang-ThinkStation-P520:~/caffe$ bash examples/mnist/create_mnist.sh 
Creating lmdb...
I1008 09:52:35.662119  6949 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_train_lmdb
I1008 09:52:35.662295  6949 convert_mnist_data.cpp:88] A total of 60000 items.
I1008 09:52:35.662302  6949 convert_mnist_data.cpp:89] Rows: 28 Cols: 28
I1008 09:52:39.572388  6949 convert_mnist_data.cpp:108] Processed 60000 files.
I1008 09:52:39.897873  6961 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_test_lmdb
I1008 09:52:39.898061  6961 convert_mnist_data.cpp:88] A total of 10000 items.
I1008 09:52:39.898067  6961 convert_mnist_data.cpp:89] Rows: 28 Cols: 28
I1008 09:52:40.528492  6961 convert_mnist_data.cpp:108] Processed 10000 files.
Done.

4.2、定义网络结构

  • 定义网络结构：hbk_mnist.prototxt
name: "hbk_mnist"

################### train/test lmdb数据层
layer {
  name: "mnist"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  transform_param {
    scale: 0.00390625
  }
  data_param {
    source: "/home/zhangkf/caffe/examples/mnist/mnist_train_lmdb"       # 训练集的位置
    batch_size: 64
    backend: LMDB
  }
}
layer {
  name: "mnist"
  type: "Data"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  transform_param {
    scale: 0.00390625
  }
  data_param {
    source: "/home/zhangkf/caffe/examples/mnist/mnist_test_lmdb"       # 测试集的位置
    batch_size: 100
    backend: LMDB
  }
}

#################### 全连接层，激活层为ReLU   784->500->10
layer {
  name: "ip1"
  type: "InnerProduct"
  bottom: "data"
  top: "ip1"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 500
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
################### 激活层
layer {
  name: "relu1"
  type: "ReLU"
  bottom: "ip1"
  top: "re1"
}
################### 全连接层
layer {
  name: "ip2"
  type: "InnerProduct"
  bottom: "re1"
  top: "ip2"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  inner_product_param {
    num_output: 10
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}

# 测试验证用，不必须，输出准确率
layer {
  name: "accuracy"
  type: "Accuracy"
  bottom: "ip2"
  bottom: "label"
  top: "accuracy"
  include {
    phase: TEST
  }
}

# 代价Cost层
layer {
  name: "loss"
  type: "SoftmaxWithLoss"
  bottom: "ip2"
  bottom: "label"
  top: "loss"
}
  • 网络结构定义完了还是不太直观，我想看看这个网络结构到底是什么样，caffe提供了一个工具这个工具 python/draw_net.py,它可以把我们定义的网络结构图形化显示出来。
python ~/caffe/python/draw_net.py  hbk_mnist.prototxt aa.png --rankdir=BT
# 其中aa.png是要生成的图片名称。
# --rankdir=BT BT表示bottom top从底向上。
# --rankdir=LR LR表示left right从左向右

4.3、配置solver文件

  • 配置solver文件：hbk_mnist_solver.prototxt
# The train/test net 文件路径
net: "hbk_mnist.prototxt"            # 网络文件

# test_iter specifies how many forward passes the test should carry out.
# In the case of MNIST, we have test batch size 100 and 100 test iterations,
# covering the full 10,000 testing images.
test_iter: 100

# 训练迭代多少次执行一次Test验证
test_interval: 500

# The base learning rate, momentum and the weight decay of the network. 默认随机梯度下降
base_lr: 0.01
momentum: 0.9
weight_decay: 0.0005

# The learning rate policy
lr_policy: "inv"
gamma: 0.0001
power: 0.75

# 多少次迭代输出一次信息
display: 100
# The maximum number of iterations
max_iter: 10001
# 存储中间结果
snapshot: 5000
snapshot_prefix: "snapshot"

# solver mode: CPU or GPU
solver_mode: GPU

4.4、评估模型执行的时间

  • 评估模型执行的时间大概是多少，运行如下命令（下面这个CPU跑的时间）：
zhangkf@wang-ThinkStation-P520:~/caffe/my_project$ caffe time -model hbk_mnist.prototxt -iterations 10
...
...
I1008 12:57:17.865840  8901 caffe.cpp:397] Average time per layer: 
I1008 12:57:17.865844  8901 caffe.cpp:400]      mnist	forward: 0.0028 ms.
I1008 12:57:17.865847  8901 caffe.cpp:403]      mnist	backward: 0.0001 ms.
I1008 12:57:17.865867  8901 caffe.cpp:400]        ip1	forward: 1.8415 ms.
I1008 12:57:17.865870  8901 caffe.cpp:403]        ip1	backward: 1.6795 ms.
I1008 12:57:17.865873  8901 caffe.cpp:400]      relu1	forward: 0.021 ms.
I1008 12:57:17.865876  8901 caffe.cpp:403]      relu1	backward: 0.022 ms.
I1008 12:57:17.865880  8901 caffe.cpp:400]        ip2	forward: 0.1029 ms.
I1008 12:57:17.865885  8901 caffe.cpp:403]        ip2	backward: 0.1368 ms.
I1008 12:57:17.865887  8901 caffe.cpp:400]       loss	forward: 0.0241 ms.
I1008 12:57:17.865890  8901 caffe.cpp:403]       loss	backward: 0.0008 ms.
I1008 12:57:17.865897  8901 caffe.cpp:408] Average Forward pass: 1.995 ms.
I1008 12:57:17.865900  8901 caffe.cpp:410] Average Backward pass: 1.8418 ms.
I1008 12:57:17.865905  8901 caffe.cpp:412] Average Forward-Backward: 3.8 ms.
I1008 12:57:17.865909  8901 caffe.cpp:414] Total Time: 38 ms.  					# 跑了38毫秒
I1008 12:57:17.865914  8901 caffe.cpp:415] *** Benchmark ends ***
  • GPU跑的时间：
zhangkf@wang-ThinkStation-P520:~/caffe/my_project$ caffe time -model hbk_mnist.prototxt -iterations 10 -gpu 0
...
...
I1008 13:00:59.563093  8943 caffe.cpp:397] Average time per layer: 
I1008 13:00:59.563104  8943 caffe.cpp:400]      mnist	forward: 0.212448 ms.
I1008 13:00:59.563109  8943 caffe.cpp:403]      mnist	backward: 0.0014048 ms.
I1008 13:00:59.563115  8943 caffe.cpp:400]        ip1	forward: 0.0407168 ms.
I1008 13:00:59.563120  8943 caffe.cpp:403]        ip1	backward: 0.0318944 ms.
I1008 13:00:59.563125  8943 caffe.cpp:400]      relu1	forward: 0.0118592 ms.
I1008 13:00:59.563130  8943 caffe.cpp:403]      relu1	backward: 0.0077504 ms.
I1008 13:00:59.563136  8943 caffe.cpp:400]        ip2	forward: 0.0259872 ms.
I1008 13:00:59.563140  8943 caffe.cpp:403]        ip2	backward: 0.02536 ms.
I1008 13:00:59.563146  8943 caffe.cpp:400]       loss	forward: 0.0810144 ms.
I1008 13:00:59.563151  8943 caffe.cpp:403]       loss	backward: 0.0143584 ms.
I1008 13:00:59.563159  8943 caffe.cpp:408] Average Forward pass: 0.41905 ms.
I1008 13:00:59.563165  8943 caffe.cpp:410] Average Backward pass: 0.113318 ms.
I1008 13:00:59.563171  8943 caffe.cpp:412] Average Forward-Backward: 0.56696 ms.
I1008 13:00:59.563177  8943 caffe.cpp:414] Total Time: 5.6696 ms. 				# 跑了5.6696毫秒
I1008 13:00:59.563191  8943 caffe.cpp:415] *** Benchmark ends ***

4.5、训练模型

zhangkf@wang-ThinkStation-P520:~/caffe/my_project$ caffe time -model hbk_mnist.prototxt -iterations 10 gpu 0
...
...
I1008 13:06:12.997189  8988 solver.cpp:347] Iteration 10000, Testing net (#0)
I1008 13:06:13.031898  9002 data_layer.cpp:73] Restarting data prefetching from start.
I1008 13:06:13.032205  8988 solver.cpp:414]     Test net output #0: accuracy = 0.9752
I1008 13:06:13.032243  8988 solver.cpp:414]     Test net output #1: loss = 0.0869556 (* 1 = 0.0869556 loss)
I1008 13:06:13.032627  8988 solver.cpp:239] Iteration 10000 (1507.71 iter/s, 0.0663259s/100 iters), loss = 0.046366
I1008 13:06:13.032657  8988 solver.cpp:258]     Train net output #0: loss = 0.0463659 (* 1 = 0.0463659 loss)
I1008 13:06:13.032671  8988 sgd_solver.cpp:112] Iteration 10000, lr = 0.00594604
I1008 13:06:13.032773  8988 solver.cpp:464] Snapshotting to binary proto file snapshot_iter_10001.caffemodel
I1008 13:06:13.037735  8988 sgd_solver.cpp:284] Snapshotting solver state to binary proto file snapshot_iter_10001.solverstate
I1008 13:06:13.040637  8988 solver.cpp:332] Optimization Done.
I1008 13:06:13.040649  8988 caffe.cpp:250] Optimization Done.
  • 我们发现程序跑的太快，一屏跟不上显示不完。下面把所有的 log 都输入一个文件里面；
  • 其中 2>&1重定向命令解释
zhangkf@wang-ThinkStation-P520:~/caffe/my_project$ caffe train -solver hbk_mnist_solver.prototxt  2>&1 | tee a.log

zhangkf@wang-ThinkStation-P520:~/caffe/my_project$ clear
zhangkf@wang-ThinkStation-P520:~/caffe/my_project$ ls
1      hbk_mnist.prototxt         snapshot_iter_10000.caffemodel   snapshot_iter_10001.caffemodel   snapshot_iter_5000.caffemodel
a.log  hbk_mnist_solver.prototxt  snapshot_iter_10000.solverstate  snapshot_iter_10001.solverstate  snapshot_iter_5000.solverstate
zhangkf@wang-ThinkStation-P520:~/caffe/my_project$

补充：vim编辑器显示行号：vim ~/.vimrc，然后写入set ts=4 set nu

4.6、分析结果

  • 根据刚才生成的 a.log 文件，分析收敛性，solver中参数设置是否合适。
  • caffe提供了一个工具，在 caffe/tools/extra目录下，parse_log.py文件，可以用它来提取有用的信息。
  • 未完待续~~~~~~~~

4.7、本地浏览器访问服务器jupyter notebook设置

本地浏览器访问服务器jupyter notebook设置

参考文献

主要参考了以下作者，在这里表示感谢！

AI大模型前沿研究
关注
专栏目录
linux ubuntu16.04 opencv3.4.2 cuda9 安装编译填坑记
Hello Word!
09-11 5676
首先说明，在此次填坑之前已经尝试了很多次在同样环境下安装opencv，包括opencv 2.4版本，均告失败。但借此可能装上了一些必要的依赖。所以这次编译的时候，只出现了这些错误。如果是初次在同样的环境下安装，可能还会报别的错误。比如，我这次就没有出现CUDA_nppi_LIBRARY (ADVANCED)的错误，可能是因为是opencv3.4版本比较高的原因，3.2有可能出现那个错误。 编译安...
Win7,Win8,Win10系统下CUDA8.0安装包CSDN下载，百度网盘下载
06-09
亲测好使，永久有效。GPU拥有大量能够并行运算且共享多级显存的处理单元，基于CUDA构架可以方便地对GPU进行访问。相比起来CPU适合逻辑运算，GPU适合数据密集的并行运算。大大降低了算法时间复杂度。
cuda8.0+cudnn6.0(win10)百度云下载链接.docx
05-02
cuda8.0+cudnn6.0(win10)百度云下载链接 版本已对应~~~~免得官网下载太慢还注册
cuda8.0+cudnn6.0+python3.5+pip_for_win10亲测可用，救急
10-31
内含cuda8.0、cudnn6.0、python3.5以及安装教程，win10亲测可用，由于文件大于110M，故上传百度云链接，大家下载下来的是百度云分享链接和密码，然后再用百度云下载即可
Debian 11 KDE Plasma桌面环境，编译Fcitx5（只编译，不安装）
庵中十三居士的博客
10-15 1870
记录了我编译fcitx5前的一些准备内容，需要安装什么工具，什么包，什么库。我安装的时候遇到的一些问题，也有所记录。
opencv---之如何安装opencv和opencv_contrib
zxyhhjs2017的博客
11-15 2702
概览 在 Ubuntu 类系统上安装 OpenCV 可以直接使用 apt 进行安装，直接在终端键入以下命令即可： $ sudo apt install libopencv-dev 1 如果只是使用 OpenCV 的基本功能，那么上面的安装方法非常简单快捷。但是如果要使用一些额外的算法，比如人脸识别，那就需要安装 opencv_contrib 来实现。本文记录了从源码安装 OpenC...
ubuntu16.04+caffe-master+cuda8.0+cudnnv5.1
03-28
里面含有ubuntu16.04+caffe-master+cuda8.0+cudnnv5.1的安装文件，太大传在网盘
Ubuntu16.04安装Cuda8.0、Cudnn5.1、Opencv3.1.0、Caffe
05-10
首先，文档标题指出了需要安装的软件版本，即Ubuntu 16.04系统中安装Cuda8.0、Cudnn5.1、Opencv3.1.0和Caffe。Cuda8.0是英伟达公司开发的并行计算平台和编程模型，它允许开发者使用英伟达GPU进行通用计算。Cudnn是...
联想拯救者(r720)Ubuntu16.04系统下安装cuda8.0+cudnn6.0+opencv+caffe+测试caffe自带MNIST数据集-附件资源
03-02
联想拯救者(r720)Ubuntu16.04系统下安装cuda8.0+cudnn6.0+opencv+caffe+测试caffe自带MNIST数据集-附件资源
ubuntu16.04安装显卡驱动+cuda8.0+cudnn5.1完整版
qq_41581769的博客
05-18 1283
反反复复折腾过好多次了，把自己的教程贴出来，希望能对大家有所帮助 一、安装显卡驱动 1.删除原有驱动，如果没装过可以忽略这一步 sudo apt-get remove --purge nvidia* 或者 sudo /usr/bin/nvidia-uninstall 卸载cuda sudo /usr/local/cuda-8.0/bin/uninstall_cuda-8.0.pl 2.禁用...
Opencv的交叉编译错误的修改
chaorenlxl的专栏
05-03 8455
第一次写博客，别喷。
Win10+cuda8.0+cudnn6.0安装过程（附百度云资源）
AYGdexiaotianxin的博客
02-15 4790
Win10+cuda8.0+cudnn6.0安装过程（附百度云资源） 从19年9月份就开始和cuda、cudnn的安装斗智斗勇，花了一周装ubuntu双系统配cuda和cudnn，结果现在主要要用win10跑程序（我的泪溢出就像雨水），装cuda8.0和cudnn6.0是为了适应caffe的环境（对，为此我还把本来装好的cuda9.0和cudnn7.1卸了，口吐芬芳^$@#￥%&……）为了...
Caffe安装记录
u012374012的专栏
03-23 2467
最近一段时间在深度学习框架caffe，看了它官方的文档和一部分代码，结合以前一些基本的神经网络知识，算是有了个大概的认识。在自己本机上ubuntu14.04上很容易配置好caffe的依赖环境，下载了一个训练好的ImageNet分类模型，写了几个python小脚本跑起来做图像分类。目前跑了10W多张，从所分类别的结果看，还不错，但要进一步验证准确性，就需要投入一些人力去辨别了，而辨别前，需要搭建一定
ubuntu 16.04 opencv3.1.0 opencvv_contrib编译遇到的问题
扯着扯着就淡淡了
04-03 3790
想搞下追踪的东西，opencv的opencv_contrib包中有类似的tracking可用 在编译好opencv3.1.0的基础上继续执行下面即可 1。下载 选择对应版本 cd /home git clone https://github.com/Itseez/opencv_contrib.git cd opencv_contrib git checkout 3.1.0 #我使用的是o...
opencv是什么？如何在CentOS7上安装使用呢？
04-09 317
OpenCV是一个基于BSD许可（开源）发行的跨平台计算机视觉库，可以运行在Linux、Windows、Android和Mac OS操作系统上。它轻量级而且高效——由一系列 C 函数和少量 C++ 类构成，同时提供了Python、Ruby、MATLAB等语言的接口，实现了图像处理和计算机视觉方面的很多通用算法。 本文采用新睿云CentOS7版本的系统上安装OpenCV，目前新睿云搞活动，云服务器...
python3-cpp扩展库[再续](private.cuda.hpp:109: error: (-216:No CUDA support))
baidu_40840693的博客
01-04 1465
python3.6(python-gcc:7.3)-anaconda-写c扩展-undefined symbol:找错误 https://blog.csdn.net/baidu_40840693/article/details/103566759 关于opencv的转换 https://github.com/Algomorph/pyboostcvconverter/blob/master/...
资源下载百度网盘（cuda8.0、cudnn5.1.10、cudnn6.0、ubuntu16.04）（已更新20180501）
热门推荐
Seven_year_Promise的博客
10-30 3万+
以下均是ubuntu系统下的文件ubuntu14.04链接：http://pan.baidu.com/s/1bpg0kbH 密码：30nw（失效，分享失败，需要的话，私信我）ubuntu16.04链接：https://pan.baidu.com/s/1NZ6OW7qnqCXYIrL-i_b_Hg 密码：yhv2cuda8.0.deb链接：https://pan.baidu.com/s/1Qe8U6...
Ubuntu16.04安装CUDA8.0+CUDNN5.1+OpenCV3.1.0+Caffe详细教程
"Ubuntu16.04安装CUDA8.0、CuDNN5.1、OpenCV3.1.0、Caffe的详细步骤，适用于单双显卡配置。" 在Ubuntu 16.04上安装CUDA、CuDNN、OpenCV和Caffe是一个涉及多个组件和依赖关系的过程。以下是一个详细的安装指南： 一...
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

AI大模型前沿研究

感谢您的打赏，我会继续努力！

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值