【caffe源码研究】第二章:使用篇(2):tools

caffe编译好后在caffe/build/tools下会生成一些工具,一一介绍使用方法。

一、caffe

这是最重要的工具

caffe: command line brew
usage: caffe <command> <args>

commands:
  train           train or finetune a model
  test            score a model
  device_query    show GPU diagnostic information
  time            benchmark model execution time

  Flags from tools/caffe.cpp:
    -gpu (Optional; run in GPU mode on given device IDs separated by ','.Use
      '-gpu all' to run on all available GPUs. The effective training batch
      size is multiplied by the number of devices.) type: string default: ""
    -iterations (The number of iterations to run.) type: int32 default: 50
    -level (Optional; network level.) type: int32 default: 0
    -model (The model definition protocol buffer text file.) type: string
      default: ""
    -phase (Optional; network phase (TRAIN or TEST). Only used for 'time'.)
      type: string default: ""
    -sighup_effect (Optional; action to take when a SIGHUP signal is received:
      snapshot, stop or none.) type: string default: "snapshot"
    -sigint_effect (Optional; action to take when a SIGINT signal is received:
      snapshot, stop or none.) type: string default: "stop"
    -snapshot (Optional; the snapshot solver state to resume training.)
      type: string default: ""
    -solver (The solver definition protocol buffer text file.) type: string
      default: ""
    -stage (Optional; network stages (not to be confused with phase), separated
      by ','.) type: string default: ""
    -weights (Optional; the pretrained weights to initialize finetuning,
      separated by ','. Cannot be set simultaneously with snapshot.)
      type: string default: ""

1. train 命令

训练模型时的命令。
常接选项是

  • solver:后接solve的配置文件
  • gpu:GPU的id号
  • weight:权重,用于tuning,caffemodel文件
  • snapshot:保存了训练中的中间状态

但是注意snapshot和weight不同时使用。

CHECK(!FLAGS_snapshot.size() || !FLAGS_weights.size())
      << "Give a snapshot to resume training or weights to finetune "
      "but not both.";

使用范例如下

GLOG_logtostderr=1 $CAFFETOOL/caffe train \
   --solver=solver_1st.prototxt \
   --weights=VGG/VGG_ILSVRC_16_layers.caffemodel \
   --gpu=0,1,2,3  2>&1 | tee log_1st.txt

翻译一下官网介绍

caffe train 可以从头学习模型,可以从快照中恢复模型,可以fine-tunes模型。
•必须的参数 -solver solver.prototxt ,配置solver文件. 
•如果是从快照中恢复,需要 -snapshot model_iter_1000.solverstate 参数
•Fine-tuning 需要 -weights model.caffemodel 参数来做模型初始化 .

例如:

# train LeNet
caffe train -solver examples/mnist/lenet_solver.prototxt

# train on GPU 2
caffe train -solver examples/mnist/lenet_solver.prototxt -gpu 2

# 从快照中恢复
caffe train -solver examples/mnist/lenet_solver.prototxt -snapshot examples/mnist/lenet_iter_5000.solverstate

# fine-tune
caffe train -solver examples/finetuning_on_flickr_style/solver.prototxt -weights models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel

2. test 命令

官网介绍

caffe test scores models by running them in the test phase and reports the net output as its score. The net architecture must be properly defined to output an accuracy measure or loss as its output. The per-batch score is reported and then the grand average is reported last.

# score the learned LeNet model on the validation set as defined in the
# model architeture lenet_train_test.prototxt
caffe test -model examples/mnist/lenet_train_test.prototxt -weights examples/mnist/lenet_iter_10000.caffemodel -gpu 0 -iterations 100

模型的输出层一定要是一个准确率或者损失层。test命令就是用训练出来的model测试TEST数据,可以修改solver文件中TEST数据达到测试其他数据集的效果。
运行结果大致如下

I1222 10:31:07.233053 27436 caffe.cpp:308] Batch 45, accuracy = 0.97
I1222 10:31:07.233103 27436 caffe.cpp:308] Batch 45, loss = 0.165972
I1222 10:31:07.301012 27436 caffe.cpp:308] Batch 46, accuracy = 0.98
I1222 10:31:07.301057 27436 caffe.cpp:308] Batch 46, loss = 0.126905
I1222 10:31:07.369249 27436 caffe.cpp:308] Batch 47, accuracy = 0.97
I1222 10:31:07.369297 27436 caffe.cpp:308] Batch 47, loss = 0.161663
I1222 10:31:07.437602 27436 caffe.cpp:308] Batch 48, accuracy = 0.99
I1222 10:31:07.437649 27436 caffe.cpp:308] Batch 48, loss = 0.0438317
I1222 10:31:07.505509 27436 caffe.cpp:308] Batch 49, accuracy = 0.98
I1222 10:31:07.505556 27436 caffe.cpp:308] Batch 49, loss = 0.0786786
I1222 10:31:07.505565 27436 caffe.cpp:313] Loss: 0.134281
I1222 10:31:07.505623 27436 caffe.cpp:325] accuracy = 0.9752
I1222 10:31:07.505653 27436 caffe.cpp:325] loss = 0.134281 (* 1 = 0.134281 loss)

3. device_query

官网介绍

caffe device_query reports GPU details for reference and checking device ordinals for running on a given device in multi-GPU machines.

# query the first device
caffe device_query -gpu 0

device_query 用于检测GPU的设备信息, 运行结果如下

[fangjin@ tools]$ ./caffe device_query -gpu 0  
I1222 11:00:09.872097 26694 caffe.cpp:138] Querying GPUs 0
I1222 11:00:10.188871 26694 common.cpp:177] Device id:                     0
I1222 11:00:10.188911 26694 common.cpp:178] Major revision number:         3
I1222 11:00:10.188916 26694 common.cpp:179] Minor revision number:         5
I1222 11:00:10.188920 26694 common.cpp:180] Name:                          Tesla K40m
I1222 11:00:10.188925 26694 common.cpp:181] Total global memory:           11995578368
I1222 11:00:10.188933 26694 common.cpp:182] Total shared memory per block: 49152
I1222 11:00:10.188938 26694 common.cpp:183] Total registers per block:     65536
I1222 11:00:10.188942 26694 common.cpp:184] Warp size:                     32
I1222 11:00:10.188946 26694 common.cpp:185] Maximum memory pitch:          2147483647
I1222 11:00:10.188951 26694 common.cpp:186] Maximum threads per block:     1024
I1222 11:00:10.188954 26694 common.cpp:187] Maximum dimension of block:    1024, 1024, 64
I1222 11:00:10.188959 26694 common.cpp:190] Maximum dimension of grid:     2147483647, 65535, 65535
I1222 11:00:10.188964 26694 common.cpp:193] Clock rate:                    745000
I1222 11:00:10.188967 26694 common.cpp:194] Total constant memory:         65536
I1222 11:00:10.188972 26694 common.cpp:195] Texture alignment:             512
I1222 11:00:10.188977 26694 common.cpp:196] Concurrent copy and execution: Yes
I1222 11:00:10.188980 26694 common.cpp:198] Number of multiprocessors:     15
I1222 11:00:10.188984 26694 common.cpp:199] Kernel execution timeout:      No

4. time

caffe time 可以测试网络时间,官网介绍如下:

caffe time benchmarks model execution layer-by-layer through timing and synchronization. This is useful to check system performance and measure relative execution times for models.

# (These example calls require you complete the LeNet / MNIST example first.)
# time LeNet training on CPU for 10 iterations
caffe time -model examples/mnist/lenet_train_test.prototxt -iterations 10

# time LeNet training on GPU for the default 50 iterations
caffe time -model examples/mnist/lenet_train_test.prototxt -gpu 0

# time a model architecture with the given weights on the first GPU for 10 iterations
caffe time -model examples/mnist/lenet_train_test.prototxt -weights examples/mnist/lenet_iter_10000.caffemodel -gpu 0 -iterations 10

实例:

#!/usr/bin/env sh
set -e
CAFFETOOL=/home/users/fangjin/caffe/build/tools

${CAFFETOOL}/caffe time --solver=lenet_solver.prototxt --model=lenet_train_test.prototxt --weights=lenet_iter_10000.caffemodel --iterations=1 --gpu=0,1,2

运行结果

I1222 11:12:06.742219 11065 caffe.cpp:361] Performing Backward
I1222 11:12:06.803246 11065 caffe.cpp:369] *** Benchmark begins ***
I1222 11:12:06.803274 11065 caffe.cpp:370] Testing for 1 iterations.
I1222 11:12:06.900956 11065 caffe.cpp:398] Iteration: 1 forward-backward time: 97 ms.
I1222 11:12:06.901000 11065 caffe.cpp:401] Average time per layer: 
I1222 11:12:06.901006 11065 caffe.cpp:404]      mnist   forward: 0.16 ms.
I1222 11:12:06.901027 11065 caffe.cpp:407]      mnist   backward: 0.001 ms.
I1222 11:12:06.901062 11065 caffe.cpp:404]      conv1   forward: 17.082 ms.
I1222 11:12:06.901072 11065 caffe.cpp:407]      conv1   backward: 17.722 ms.
I1222 11:12:06.901080 11065 caffe.cpp:404]      pool1   forward: 7.324 ms.
I1222 11:12:06.901088 11065 caffe.cpp:407]      pool1   backward: 3.516 ms.
I1222 11:12:06.901096 11065 caffe.cpp:404]      conv2   forward: 15.294 ms.
I1222 11:12:06.901104 11065 caffe.cpp:407]      conv2   backward: 28.055 ms.
I1222 11:12:06.901111 11065 caffe.cpp:404]      pool2   forward: 3.847 ms.
I1222 11:12:06.901119 11065 caffe.cpp:407]      pool2   backward: 2.194 ms.
I1222 11:12:06.901126 11065 caffe.cpp:404]        ip1   forward: 0.742 ms.
I1222 11:12:06.901134 11065 caffe.cpp:407]        ip1   backward: 1.185 ms.
I1222 11:12:06.901141 11065 caffe.cpp:404]      relu1   forward: 0.057 ms.
I1222 11:12:06.901149 11065 caffe.cpp:407]      relu1   backward: 0.117 ms.
I1222 11:12:06.901156 11065 caffe.cpp:404]        ip2   forward: 0.101 ms.
I1222 11:12:06.901165 11065 caffe.cpp:407]        ip2   backward: 0.114 ms.
I1222 11:12:06.901171 11065 caffe.cpp:404]       loss   forward: 0.105 ms.
I1222 11:12:06.901180 11065 caffe.cpp:407]       loss   backward: 0.004 ms.
I1222 11:12:06.901190 11065 caffe.cpp:412] Average Forward pass: 44.719 ms.
I1222 11:12:06.901197 11065 caffe.cpp:414] Average Backward pass: 52.923 ms.
I1222 11:12:06.901206 11065 caffe.cpp:416] Average Forward-Backward: 97 ms.
I1222 11:12:06.901213 11065 caffe.cpp:418] Total Time: 97 ms.
I1222 11:12:06.901219 11065 caffe.cpp:419] *** Benchmark ends ***

5. 其他

注意GPU这个参数项

-gpu (Optional; run in GPU mode on given device IDs separated by ','.Use
      '-gpu all' to run on all available GPUs. The effective training batch
      size is multiplied by the number of devices.) type: string default: ""

接多个GPU时,使用逗号隔开,如'-gpu 0,1,2',全部GPU则使用’-gpu all

参数项可以写-gpu 0或者--gpu=0,两种写法都可以。

二、compute_image_mean

这个命令用于计算图像均值。

在前面介绍可以看到,使用了一个

MEAN_VALUE = 128

在caffe可以先计算均值,然后对所有图像去除均值,在上文中是将均值写成固定的128,现在展示如何使用caffe自己编译的工具计算均值。

compute_image_mean: Compute the mean_image of a set of images given by a leveldb/lmdb
Usage:
    compute_image_mean [FLAGS] INPUT_DB [OUTPUT_FILE]


  Flags from tools/compute_image_mean.cpp:
    -backend (The backend {leveldb, lmdb} containing the images) type: string
      default: "lmdb"

写脚本compute_image_mean.sh

#!/usr/bin/env sh
set -e

CAFFETOOL=/home/users/fangjin/caffe/build/tools

${CAFFETOOL}/compute_image_mean number_train_lmdb image_mean.binaryproto  

就会生成均值文件image_mean.binaryproto
使用方式是将Data层的transform_param添加一个mean_file,训练集和验证集都要添加

transform_param {
    scale: 0.00390625
    mean_file: "F:/caffe/data/image_mean.binaryproto"
  }

然后同样的方式训练。

三、convert_imageset

convert_imageset: Convert a set of images to the leveldb/lmdb
format used as input for Caffe.
Usage:
    convert_imageset [FLAGS] ROOTFOLDER/ LISTFILE DB_NAME
The ImageNet dataset for the training demo is at
    http://www.image-net.org/download-images


  Flags from tools/convert_imageset.cpp:
    -backend (The backend {lmdb, leveldb} for storing the result) type: string
      default: "lmdb"
    -check_size (When this option is on, check that all the datum have the same
      size) type: bool default: false
    -encode_type (Optional: What type should we encode the image as
      ('png','jpg',...).) type: string default: ""
    -encoded (When this option is on, the encoded image will be save in datum)
      type: bool default: false
    -gray (When this option is on, treat images as grayscale ones) type: bool
      default: false
    -resize_height (Height images are resized to) type: int32 default: 0
    -resize_width (Width images are resized to) type: int32 default: 0
    -shuffle (Randomly shuffle the order of images and their labels) type: bool
      default: false

这个命令的运行方法在【caffe源码研究】第二章:使用篇(1): 制作数据集中已经介绍过了。
代码示例

TOOLS=/home/users/fangjin/caffe/build/tools                                             
ESIZE_HEIGHT=32
RESIZE_WIDTH=32
TRAIN_DATA_ROOT=/home/users/fangjin/test/number_data/

echo "Creating train lmdb..."
GLOG_logtostderr=1 $TOOLS/convert_imageset \
   --resize_height=32 \
   --resize_width=32 \
   --shuffle \
   $TRAIN_DATA_ROOT \
   train.txt \
   number_train_lmdb

echo "Creating test lmdb..."
GLOG_logtostderr=1 $TOOLS/convert_imageset \
   --resize_height=32 \
   --resize_width=32 \
   --shuffle \
   $TRAIN_DATA_ROOT \
   test.txt \
   number_test_lmdb  #输出

四、extract_features

This program takes in a trained network and an input data layer, and then extract features of the input data produced by the net.
Usage: extract_features  pretrained_net_param  feature_extraction_proto_file  extract_feature_blob_name1[,name2,...]  save_feature_dataset_name1[,name2,...]  num_mini_batches  db_type  [CPU/GPU] [DEVICE_ID=0]
Note: you can extract multiple features in one pass by specifying multiple feature blob names and dataset names separated by ','. The names cannot contain white space characters and the number of blobs and datasets must be equal.

就是一个C++的接口用于提取特征,现实中作用可能并没那么大,现在很多都是使用python接口,比较方便。

官网上提供了一个例子。

先建立一个临时文件

mkdir examples/_temp

将图片路径写入文本

find `pwd`/examples/images -type f -exec echo {} \; > examples/_temp/temp.txt

temp.txt文件内容如下

/home/users/fangjin/caffe/examples/images/cat gray.jpg                                  
/home/users/fangjin/caffe/examples/images/cat.jpg
/home/users/fangjin/caffe/examples/images/cat_gray.jpg
/home/users/fangjin/caffe/examples/images/fish-bike.jpg

将标签也写入文本,这里统一用0

sed "s/$/ 0/" examples/_temp/temp.txt > examples/_temp/file_list.txt

file_list.txt内容如下

/home/users/fangjin/caffe/examples/images/cat gray.jpg 0                                  
/home/users/fangjin/caffe/examples/images/cat.jpg 0
/home/users/fangjin/caffe/examples/images/cat_gray.jpg 0
/home/users/fangjin/caffe/examples/images/fish-bike.jpg 0

用脚本./data/ilsvrc12/get_ilsvrc_aux.sh下载 ILSVRC 的均值。

我们使用data/ilsvrc212/imagenet_mean.binaryproto 和对应的prototxt文件.

拷贝一份imagenet_val.prototxt

cp examples/feature_extraction/imagenet_val.prototxt examples/_temp

现在开始运行脚本


./build/tools/extract_features.bin models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel examples/_temp/imagenet_val.prototxt fc7 examples/_temp/features 10 leveldb

这里提取的是fc7这一层的特征,也可以使用conv5,pool3等。最后的数字参数是mini-batched的个数,特征存储于examples/_temp/features中,leveldb格式。

[fangjin@ caffe]$ ./build/tools/extract_features.bin models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel examples/_temp/imagenet_val.prototxt fc7 examples/_temp/features 10 leveldb
E1222 14:11:18.487232 14915 extract_features.cpp:62] Using CPU
E1222 14:11:19.759531 14915 extract_features.cpp:133] Extracting Features
E1222 14:11:59.131815 14915 extract_features.cpp:176] Extracted features of 500 query images for feature blob fc7
E1222 14:11:59.132774 14915 extract_features.cpp:181] Successfully extracted the features!          

五、其他

其他的一些命令,例如finetune_net等已经被抛弃了,统一在caffe命令接口里。剩下的三个也是用于把老的结构换成新的结构,一般用不到。

  • upgrade_net_proto_binary.bin
  • upgrade_net_proto_text.bin
  • upgrade_solver_proto_text.bin
  • 3
    点赞
  • 7
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值