一.安装
1.1 安装Matio
$ tar zxf matio-1.5.2.tar.gz
$ cd matio-1.5.2
$ ./configure
$ make
$ make check
$ make install
$ sudo ldconfig
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /home/baw010/houqiqi/dependencies/matio-1.5.2/src
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /home/baw010/houqiqi/dependencies/matio-1.5.2/src/.libs
在.bashrc中添加路径
export LD_LIBRARY_PATH=/your/path/to/libmatio.so.2
source ~/.bashrc
重启服务器
出错:运行./configure时出错如下:
sudo apt-get automake
二、下载Deeplabv2并编译
1、下载代码:
git clone https://github.com/xmojiao/deeplab_v2.git
2、对 caffe 进行编译:
1、安装依赖项
udo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
2、根据deeplab-public-ver2/python目录下的requirements.txt安装python附加依赖库:
for req in $(cat requirements.txt); do sudo pip install $req; done
3、根据需求调整Makefile.config文件(尽量少改动,修改前记得备份)
USE_CUDNN:=1
使用cudnn加速,取消这行注释
如果使用2.4(没有装opencv3)的话就不用管OPENCV_VERSION
CUDA_DIR := /usr/local/cuda
如果装的是默认位置就不用改 CUDA_ARCH :=
这里删掉5以下的(官网查询1080Ti显卡的计算性能是6.1:https://developer.nvidia.com/cuda-gpus) BLAS :=atlas
选择blas版本,这里按默认选择atlas(如果以后需要使用intel mkl可以在这里修改后再次编译caffe) PYTHON_INCLUDE :=
这里默认是/user/lib/...
但是pip把numpy实际安装在了/usr/local/lib/...
需要检查后改一下,同 时PYTHON_LIB
后也要追加/usr/local/lib/
目录 MATLAB_DIR:= /usr/local/MATLAB/R2015b
这里改为matlab安装路径
INCLUDE_DIRS
这里追加 /usr/include/hdf5/serial
同时打开Makefile在LIBRARIES +=
后把hdf5_hl hdf5
替换成hdf5_serial_hl hdf5_serial
4、编译caffe
make all -j4
make pycaffe
出错:
解决办法:打开./include/caffe/common.cuh文件,在automaticAdd前添加宏判断即可:
#if !defined(__CUDA_ARCH__) || __CUDA_ARCH__ >= 600
#else
static __inline__ __device__ double atomicAdd(double* address, double val)
{ ...
}
#endif
2、对 run_pascal.sh 进行调试:
(1)首先准备好数据
文章中使用的数据集并不全是pascal-voc2012,而是由voc2012和另外一个数据集合并而成
数据下载
# augmented PASCAL VOC
mkdir -p ~/DL_dataset
cd ~/DL_dataset #save datasets 为$DATASETS
wget http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz # 1.3 GB
tar -zxvf benchmark.tgz
mv benchmark_RELEASE VOC_aug
# original PASCAL VOC 2012
wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar # 2 GB
tar -xvf VOCtrainval_11-May-2012.tar
mv VOCdevkit/VOC2012 VOC2012_orig && rm -r VOCdevkit
因为pascal voc2012增强数据集的label是mat格式的文件,所以我们需要把mat格式的label转为png格式的图片.
cd ~/DL_dataset/VOC_aug/dataset
mkdir cls_png
cd ~/deeplab_v2/voc2012/
./mat2png.py ~/DL_dataset/VOC_aug/dataset/cls ~/DL_dataset/VOC_aug/dataset/cls_png
pascal voc2012原始数据集的label为三通道RGB图像,但是caffe最后一层softmax loss 层只能识别一通道的label,所以此处我们需要对原始数据集的label进行降维
cd ~/DL_dataset/VOC2012_orig
mkdir SegmentationClass_1D
cd ~/deeplab_v2/voc2012
./convert_labels.py ~/DL_dataset/VOC2012_orig/SegmentationClass/ ~/DL_dataset/VOC2012_orig/ImageSets/Segmentation/trainval.txt ~/DL_dataset/VOC2012_orig/SegmentationClass_1D/
(3)数据融合
此时已经处理好好pascal voc2012 增强数据集和pascal voc2012的原始数据集,为了便于train.txt等文件的调用,将两个文件夹数据合并到同一个文件中.现有文件目录如下:
现分别pascal voc2012增强数据集里的images和labels复制到增强数据集中,若重复则覆盖,合将并数据集的操作如下:
cp ~/Desktop/my_dataset/VOC2012_orig/SegmentationClass_1D/* ~/Desktop/my_dataset/VOC_aug/dataset/cls_png cp ~/Desktop/my_dataset/VOC2012_orig/JPEGImages/* ~/Desktop/my_dataset/VOC_aug/dataset/img/
(4)文件名修改
对应 train.txt文件 的数据集文件名,修改文件名。
cd ~/DL_dataset/VOC_aug/dataset mv ./img ./JPEGImages mv ./cls_png ./SegmentationClassAug
(5)修改并运行 run_pascal.sh
1)准备必要的文件
需要的文件从这里下载 deeplabv2 有两种模型(vgg,Res102),我用的vgg ,http://liangchiehchen.com/projects/DeepLab_Models.html
总共需要的文件如图所示:
Desktop/deeplab_v2/voc2012/model/deeplab_largeFOV 里没有model,需要把下载好的vgg的model放入文件,如图所示:
至此,所有需要的文件全部完毕。
2)运行 train 和 test
进入/Desktop/deeplab_v2/voc2012,修改 run_pascal.sh 文件,主要是修改路径,我的修改后的文件如下:
接下来运行代码:#!/bin/sh ## MODIFY PATH for YOUR SETTING ROOT_DIR=/home/mmt/Desktop/my_dataset CAFFE_DIR=/home/mmt/Desktop/deeplab_v2/deeplab-public-ver2 CAFFE_BIN=${CAFFE_DIR}/build/tools/caffe.bin EXP=. if [ "${EXP}" = "." ]; then NUM_LABELS=21 DATA_ROOT=${ROOT_DIR}/VOC_aug/dataset/ else NUM_LABELS=0 echo "Wrong exp name" fi ## Specify which model to train ########### voc12 ################ NET_ID=deeplab_largeFOV ## Variables used for weakly or semi-supervisedly training #TRAIN_SET_SUFFIX= TRAIN_SET_SUFFIX=_aug #TRAIN_SET_STRONG=train #TRAIN_SET_STRONG=train200 #TRAIN_SET_STRONG=train500 #TRAIN_SET_STRONG=train1000 #TRAIN_SET_STRONG=train750 #TRAIN_SET_WEAK_LEN=5000 DEV_ID=0 ##### ## Create dirs CONFIG_DIR=${EXP}/config/${NET_ID} MODEL_DIR=${EXP}/model/${NET_ID} mkdir -p ${MODEL_DIR} LOG_DIR=${EXP}/log/${NET_ID} mkdir -p ${LOG_DIR} export GLOG_log_dir=${LOG_DIR} ## Run RUN_TRAIN=1 #1时train RUN_TEST=0 #1时test RUN_TRAIN2=0 RUN_TEST2=0 ## Training #1 (on train_aug) if [ ${RUN_TRAIN} -eq 1 ]; then # LIST_DIR=${EXP}/list TRAIN_SET=train${TRAIN_SET_SUFFIX} if [ -z ${TRAIN_SET_WEAK_LEN} ]; then TRAIN_SET_WEAK=${TRAIN_SET}_diff_${TRAIN_SET_STRONG} comm -3 ${LIST_DIR}/${TRAIN_SET}.txt ${LIST_DIR}/${TRAIN_SET_STRONG}.txt > ${LIST_DIR}/${TRAIN_SET_WEAK}.txt else TRAIN_SET_WEAK=${TRAIN_SET}_diff_${TRAIN_SET_STRONG}_head${TRAIN_SET_WEAK_LEN} comm -3 ${LIST_DIR}/${TRAIN_SET}.txt ${LIST_DIR}/${TRAIN_SET_STRONG}.txt | head -n ${TRAIN_SET_WEAK_LEN} > ${LIST_DIR}/${TRAIN_SET_WEAK}.txt fi # MODEL=${EXP}/model/${NET_ID}/init.caffemodel # echo Training net ${EXP}/${NET_ID} for pname in train solver; do sed "$(eval echo $(cat sub.sed))" \ ${CONFIG_DIR}/${pname}.prototxt > ${CONFIG_DIR}/${pname}_${TRAIN_SET}.prototxt done CMD="${CAFFE_BIN} train \ --solver=${CONFIG_DIR}/solver_${TRAIN_SET}.prototxt \ --gpu=${DEV_ID}" if [ -f ${MODEL} ]; then CMD="${CMD} --weights=${MODEL}" fi echo Running ${CMD} && ${CMD} fi ## Test #1 specification (on val or test) if [ ${RUN_TEST} -eq 1 ]; then # for TEST_SET in val; do TEST_ITER=`cat ${EXP}/list/${TEST_SET}.txt | wc -l` MODEL=${EXP}/model/${NET_ID}/test.caffemodel if [ ! -f ${MODEL} ]; then MODEL=`ls -t ${EXP}/model/${NET_ID}/train_iter_*.caffemodel | head -n 1` fi # echo Testing net ${EXP}/${NET_ID} FEATURE_DIR=${EXP}/features/${NET_ID} mkdir -p ${FEATURE_DIR}/${TEST_SET}/fc8 mkdir -p ${FEATURE_DIR}/${TEST_SET}/fc9 mkdir -p ${FEATURE_DIR}/${TEST_SET}/seg_score sed "$(eval echo $(cat sub.sed))" \ ${CONFIG_DIR}/test.prototxt > ${CONFIG_DIR}/test_${TEST_SET}.prototxt CMD="${CAFFE_BIN} test \ --model=${CONFIG_DIR}/test_${TEST_SET}.prototxt \ --weights=${MODEL} \ --gpu=${DEV_ID} \ --iterations=${TEST_ITER}" echo Running ${CMD} && ${CMD} done fi ## Training #2 (finetune on trainval_aug) if [ ${RUN_TRAIN2} -eq 1 ]; then # LIST_DIR=${EXP}/list TRAIN_SET=trainval${TRAIN_SET_SUFFIX} if [ -z ${TRAIN_SET_WEAK_LEN} ]; then TRAIN_SET_WEAK=${TRAIN_SET}_diff_${TRAIN_SET_STRONG} comm -3 ${LIST_DIR}/${TRAIN_SET}.txt ${LIST_DIR}/${TRAIN_SET_STRONG}.txt > ${LIST_DIR}/${TRAIN_SET_WEAK}.txt else TRAIN_SET_WEAK=${TRAIN_SET}_diff_${TRAIN_SET_STRONG}_head${TRAIN_SET_WEAK_LEN} comm -3 ${LIST_DIR}/${TRAIN_SET}.txt ${LIST_DIR}/${TRAIN_SET_STRONG}.txt | head -n ${TRAIN_SET_WEAK_LEN} > ${LIST_DIR}/${TRAIN_SET_WEAK}.txt fi # MODEL=${EXP}/model/${NET_ID}/init2.caffemodel if [ ! -f ${MODEL} ]; then MODEL=`ls -t ${EXP}/model/${NET_ID}/train_iter_*.caffemodel | head -n 1` fi # echo Training2 net ${EXP}/${NET_ID} for pname in train solver2; do sed "$(eval echo $(cat sub.sed))" \ ${CONFIG_DIR}/${pname}.prototxt > ${CONFIG_DIR}/${pname}_${TRAIN_SET}.prototxt done CMD="${CAFFE_BIN} train \ --solver=${CONFIG_DIR}/solver2_${TRAIN_SET}.prototxt \ --weights=${MODEL} \ --gpu=${DEV_ID}" echo Running ${CMD} && ${CMD} fi ## Test #2 on official test set if [ ${RUN_TEST2} -eq 1 ]; then # for TEST_SET in val test; do TEST_ITER=`cat ${EXP}/list/${TEST_SET}.txt | wc -l` MODEL=${EXP}/model/${NET_ID}/test2.caffemodel if [ ! -f ${MODEL} ]; then MODEL=`ls -t ${EXP}/model/${NET_ID}/train2_iter_*.caffemodel | head -n 1` fi # echo Testing2 net ${EXP}/${NET_ID} FEATURE_DIR=${EXP}/features2/${NET_ID} mkdir -p ${FEATURE_DIR}/${TEST_SET}/fc8 mkdir -p ${FEATURE_DIR}/${TEST_SET}/crf sed "$(eval echo $(cat sub.sed))" \ ${CONFIG_DIR}/test.prototxt > ${CONFIG_DIR}/test_${TEST_SET}.prototxt CMD="${CAFFE_BIN} test \ --model=${CONFIG_DIR}/test_${TEST_SET}.prototxt \ --weights=${MODEL} \ --gpu=${DEV_ID} \ --iterations=${TEST_ITER}" echo Running ${CMD} && ${CMD} done fi
运行结果如下:~/Desktop/deeplab_v2/voc2012 sh ./run_pascal.sh
Test:
将相应变量改为1:
运行结果如下:~/Desktop/deeplab_v2/voc2012 sh ./run_pascal.sh
因为结果保存的是mat文件,如果想转换成png的形式,运行:
修改create_labels_21.py的路径(若只想分割出一类物体,则只需把create_labels_21.py中的palette的其它类别的像素值修改为0即可),然后此目录运行:cd ~/Desktop/deeplab_v2/voc2012
python create_labels_21.py
(6)修改并运行 run_densecrf.sh
1)首先对densecrf进行编译。
2)数据整理cd ~/Desktop/deeplab_v2/deeplab-public-ver2/densecrf/ make
因为densecrf只识别ppm格式的图像,所以要转换格式。进入/Desktop/deeplab_v2/deeplab-public-ver2/densecrf/my_script,里面有自带的修改ppm 的MATLAB程序,修改路径,直接运行即可。
代码如下:
3)接下来,修改 run_densecrf.sh, 注意把 MODEL_NAME=deeplab_largeFOV修改了,原文件少了一个 p 。% save jpg images as bin file for cpp % is_server = 1; dataset = 'voc2012'; %'coco', 'voc2012' if is_server if strcmp(dataset, 'voc2012') img_folder = '/home/mmt/Desktop/my_dataset/VOC_aug/dataset/JPEGImages' save_folder = '/home/mmt/Desktop/my_dataset/VOC_aug/dataset/PPMImages'; elseif strcmp(dataset, 'coco') img_folder = '/rmt/data/coco/JPEGImages'; save_folder = '/rmt/data/coco/PPMImages'; end else img_folder = '../img'; save_folder = '../img_ppm'; end if ~exist(save_folder, 'dir') mkdir(save_folder); end img_dir = dir(fullfile(img_folder, '*.jpg')); for i = 1 : numel(img_dir) fprintf(1, 'processing %d (%d)...\n', i, numel(img_dir)); img = imread(fullfile(img_folder, img_dir(i).name)); img_fn = img_dir(i).name(1:end-4); save_fn = fullfile(save_folder, [img_fn, '.ppm']); imwrite(img, save_fn); end
DATASET=voc2012 修改; SAVE_DIR=/home/mmt/Desktop/deeplab_v2/${DATASET}/res/${FEATURE_NAME}/${MODEL_NAME}/${TEST_SET} 修改; CRF_DIR=/home/mmt/Desktop/deeplab_v2/deeplab-public-ver2/densecrf 修改; if [ ${DATASET} == "voc2012" ] then IMG_DIR_NAME=VOC_aug/dataset 修改; FEATURE_DIR=/home/mmt/Desktop/deeplab_v2/${DATASET}/${FEATURE_NAME}/${MODEL_NAME}/${TEST_SET}/${FEATURE_TYPE} 修改; 同时把一些不需要的语句都注释掉,要不然容易出错,显示找不到文件。 修改后的文件如下: #!/bin/bash ########################################### # You can either use this script to generate the DenseCRF post-processed results # or use the densecrf_layer (wrapper) in Caffe ########################################### DATASET=voc2012 LOAD_MAT_FILE=1 MODEL_NAME=deeplab_largeFOV TEST_SET=val #val, test # the features folder save the features computed via the model trained with the train set # the features2 folder save the features computed via the model trained with the trainval set FEATURE_NAME=features #features, features2 FEATURE_TYPE=fc8 # specify the parameters MAX_ITER=10 Bi_W=4 Bi_X_STD=49 Bi_Y_STD=49 Bi_R_STD=5 Bi_G_STD=5 Bi_B_STD=5 POS_W=3 POS_X_STD=3 POS_Y_STD=3 ####################################### # MODIFY THE PATY FOR YOUR SETTING ####################################### SAVE_DIR=/home/mmt/Desktop/deeplab_v2/${DATASET}/res/${FEATURE_NAME}/${MODEL_NAME}/${TEST_SET}/${FEATURE_TYPE}/post_densecrf_W${Bi_W}_XStd${Bi_X_STD}_RStd${Bi_R_STD}_PosW${POS_W}_PosXStd${POS_X_STD} echo "SAVE TO ${SAVE_DIR}" CRF_DIR=/home/mmt/Desktop/deeplab_v2/deeplab-public-ver2/densecrf #if [ ${DATASET} == "voc2012" ] #then IMG_DIR_NAME=VOC_aug/dataset #elif [ ${DATASET} == "coco" ] #then # IMG_DIR_NAME=coco #elif [ ${DATASET} == "voc10_part" ] #then # IMG_DIR_NAME=pascal/VOCdevkit/VOC2012 #fi # NOTE THAT the densecrf code only loads ppm images IMG_DIR=/home/mmt/Desktop/my_dataset/${IMG_DIR_NAME}/PPMImages #if [ ${LOAD_MAT_FILE} == 1 ] #then # the features are saved in .mat format CRF_BIN=${CRF_DIR}/prog_refine_pascal_v4 FEATURE_DIR=/home/mmt/Desktop/deeplab_v2/${DATASET}/${FEATURE_NAME}/${MODEL_NAME}/${TEST_SET}/${FEATURE_TYPE} #else # the features are saved in .bin format (has called SaveMatAsBin.m in the densecrf/my_script) # CRF_BIN=${CRF_DIR}/prog_refine_pascal # FEATURE_DIR=/home/mmt/Desktop/deeplab_v2/${DATASET}/${FEATURE_NAME}/${MODEL_NAME}/${TEST_SET}/${FEATURE_TYPE}/bin #fi mkdir -p ${SAVE_DIR} # run the program ${CRF_BIN} -id ${IMG_DIR} -fd ${FEATURE_DIR} -sd ${SAVE_DIR} -i ${MAX_ITER} -px ${POS_X_STD} -py ${POS_Y_STD} -pw ${POS_W} -bx ${Bi_X_STD} -by ${Bi_Y_STD} -br ${Bi_R_STD} -bg ${Bi_G_STD} -bb ${Bi_B_STD} -bw ${Bi_W}
进入文件路径,运行程序,结果如下图:
cd ~/Desktop/deeplab_v2/voc2012/ sh sh ./run_densecrf.sh
4)然后运行 /home/mmt/crf/deeplab-public-ver2/densecrf/my_script/GetDenseCRFResult.m把bin生成图片格式
注意修改文件路径(GetDenseCRFResult.m,SetupEnv在/deeplab_v2/deeplab-public-ver2/matlab/my_script中),
两个程序的代码如下:
GetDenseCRFResult.m % compute the densecrf result (.bin) to png % addpath('/home/mmt/Desktop/deeplab_v2/deeplab-public-ver2/matlab/my_script'); SetupEnv; %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % You do not need to change values below %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% if is_server if learn_crf post_folder = sprintf('post_densecrf_W%d_XStd%d_RStd%d_PosW%d_PosXStd%d_ModelType%d_Epoch%d', bi_w, bi_x_std, bi_r_std, pos_w, pos_x_std, model_type, epoch); map_folder = fullfile('/home/mmt/Desktop/deeplab_v2', dataset, 'densecrf', 'res', feature_name, model_name, testset, feature_type, post_folder); save_root_folder = fullfile('/home/mmt/Desktop/deeplab_v2', dataset, 'res', feature_name, model_name, testset, feature_type, post_folder); ; else post_folder = sprintf('post_densecrf_W%d_XStd%d_RStd%d_PosW%d_PosXStd%d', bi_w, bi_x_std, bi_r_std, pos_w, pos_x_std); map_folder = fullfile('/home/mmt/Desktop/deeplab_v2', dataset, 'res', feature_name, model_name, testset, feature_type, post_folder); save_root_folder = map_folder; end else map_folder = '../result'; end map_dir = dir(fullfile(map_folder, '*.bin')); fprintf(1,' saving to %s\n', save_root_folder); if strcmp(dataset, 'voc2012') seg_res_dir = [save_root_folder '/results/VOC2012/']; elseif strcmp(dataset, 'coco') seg_res_dir = [save_root_folder, '/results/COCO2014/']; else error('Wrong dataset!'); end save_result_folder = fullfile(seg_res_dir, 'Segmentation', [id '_' testset '_cls']); if ~exist(save_result_folder, 'dir') mkdir(save_result_folder); end for i = 1 : numel(map_dir) fprintf(1, 'processing %d (%d)...\n', i, numel(map_dir)); map = LoadBinFile(fullfile(map_folder, map_dir(i).name), 'int16'); img_fn = map_dir(i).name(1:end-4); imwrite(uint8(map), colormap, fullfile(save_result_folder, [img_fn, '.png'])); end
SetupEnv.m % set up the environment variables % clear all; close all; load('./pascal_seg_colormap.mat'); is_server = 1; crf_load_mat = 1; % the densecrf code load MAT files directly (no call SaveMatAsBin.m) % used ONLY by DownSampleFeature.m learn_crf = 0; % NOT USED. Set to 0 is_mat = 1; % the results to be evaluated are saved as mat (1) or png (0) has_postprocess = 0; % has done densecrf post processing (1) or not (0) is_argmax = 0; % the output has been taken argmax already (e.g., coco dataset). % assume the argmax takes C-convention (i.e., start from 0) debug = 0; % if debug, show some results % vgg128_noup (not optimized well), aka DeepLab % bi_w = 5, bi_x_std = 50, bi_r_std = 10 % vgg128_ms_pool3, aka DeepLab-MSc % bi_w = 3, bi_x_std = 95, bi_r_std = 3 % vgg128_noup_pool3_cocomix, aka DeepLab-COCO % bi_w = 5, bi_x_std = 67, bi_r_std = 3 %% these are used for the bounding box weak annotation experiments (i.e., to generate the Bbox-Seg) % erode_gt (bbox) % bi_w = 41, bi_x_std = 33, bi_r_std = 4 % erode_gt/bboxErode20 % bi_w = 45, bi_x_std = 37, bi_r_std = 3, pos_w = 15, pos_x_std = 3 % % initial or default values for crf %% 这几个参数要修改与run_densecrf.sh中的一致。 bi_w = 4; bi_x_std = 49; bi_r_std = 5; pos_w = 3; pos_x_std = 3; % dataset = 'voc2012'; %'voc12', 'coco' 修改 trainset = 'train_aug'; % not used testset = 'val'; %'val', 'test' model_name = 'deeplab_largeFOV'; % 修改 feature_name = 'features'; feature_type = 'fc8'; % fc8 / crf id = 'comp6'; %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% used for cross-validation %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% rng(10) % downsampling files for cross-validation down_sample_method = 2; % 1: equally sample with "down_sample_rate", 2: randomly pick "num_sample" samples down_sample_rate = 8; num_sample = 100; % number of samples used for cross-validation % ranges for cross-validation range_pos_w = [3]; range_pos_x_std = [3]; range_bi_w = [5]; range_bi_x_std = [49]; range_bi_r_std = [4 5];
5)如果只想分割出一类物体,只需要把pascal_seg_colormap中的其它类别的像素值全改成0就行了。
6)至此,deeplabv2 程序已调试完。