caffe示例实现之5用CaffeNet训练与测试自己的数据集

转自：http://blog.csdn.net/liumaolincycle/article/details/48475479

本文主要来自Caffe作者Yangqing Jia网站给出的examples。

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@article</span>{jia2014caffe,
  <span class="hljs-constant" style="box-sizing: border-box;">Author</span> = {<span class="hljs-constant" style="box-sizing: border-box;">Jia</span>, <span class="hljs-constant" style="box-sizing: border-box;">Yangqing</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <span class="hljs-constant" style="box-sizing: border-box;">Shelhamer</span>, <span class="hljs-constant" style="box-sizing: border-box;">Evan</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <span class="hljs-constant" style="box-sizing: border-box;">Donahue</span>, <span class="hljs-constant" style="box-sizing: border-box;">Jeff</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <span class="hljs-constant" style="box-sizing: border-box;">Karayev</span>, <span class="hljs-constant" style="box-sizing: border-box;">Sergey</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <span class="hljs-constant" style="box-sizing: border-box;">Long</span>, <span class="hljs-constant" style="box-sizing: border-box;">Jonathan</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <span class="hljs-constant" style="box-sizing: border-box;">Girshick</span>, <span class="hljs-constant" style="box-sizing: border-box;">Ross</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <span class="hljs-constant" style="box-sizing: border-box;">Guadarrama</span>, <span class="hljs-constant" style="box-sizing: border-box;">Sergio</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <span class="hljs-constant" style="box-sizing: border-box;">Darrell</span>, <span class="hljs-constant" style="box-sizing: border-box;">Trevor</span>},
  <span class="hljs-constant" style="box-sizing: border-box;">Journal</span> = {arXiv preprint <span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">arXiv:</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1408.5093</span>},
  <span class="hljs-constant" style="box-sizing: border-box;">Title</span> = {<span class="hljs-constant" style="box-sizing: border-box;">Caffe</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:</span> <span class="hljs-constant" style="box-sizing: border-box;">Convolutional</span> <span class="hljs-constant" style="box-sizing: border-box;">Architecture</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-constant" style="box-sizing: border-box;">Fast</span> <span class="hljs-constant" style="box-sizing: border-box;">Feature</span> <span class="hljs-constant" style="box-sizing: border-box;">Embedding</span>},
  <span class="hljs-constant" style="box-sizing: border-box;">Year</span> = {<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2014</span>}
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>

1.准备数据

该教程主要是教我们在自己的数据上训练模型，caffe的github上已经提供了训练好的CaffeNet模型，旧地址已经不在了，新地址是model zoo。 
这里的ImageNet指的是ILSVRC12挑战赛中用到的数据集，当然也可以在整个ImageNet上训练，只不过硬盘容量要更大一点，训练时间要更久一点。这里我打算只用其中一部分数据来训练，全部数据实在太大。 
该教程假设已经下载好了ImageNet的训练数据和验证数据，需要一些额外的文件，用下面的命令下载

<code class="language-shell hljs haskell has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">./<span class="hljs-typedef" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">data</span>/ilsvrc12/get_ilsvrc_aux.sh</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

然后就会看到这个文件夹里多了好多文件，训练和测试的输入是用train.txt和val.txt描述的，以文本形式列出所有文件及标签，注意这里使用与ILSVRC工具集不同的标签索引：按ASCII码顺序对类名进行排序，然后依次标注为0~999（注意是从0开始的），synset_words.txt中是synset（eg：n01440764）与类别名的映射。 
模仿他们的做法，我现在只保留了n03791053摩托车、n03895866客车、n04285008跑车、n04467665大货车、n04487081无轨电车这5类。

train.txt和val.txt都是下面这样的格式：

看一下examples/imagenet/create_imagenet.sh文件，在example/imagenet下新建一个文件夹，名为myself，把所有要用到的图片和脚本放进去，包括create_imagenet.sh，我们在这里面修改它：

<code class="language-makefile hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-constant" style="box-sizing: border-box;">EXAMPLE</span>=examples/imagenet/myself
<span class="hljs-constant" style="box-sizing: border-box;">DATA</span>=examples/imagenet/myself
<span class="hljs-constant" style="box-sizing: border-box;">TOOLS</span>=build/tools

<span class="hljs-constant" style="box-sizing: border-box;">TRAIN_DATA_ROOT</span>=examples/imagenet/myself/
<span class="hljs-constant" style="box-sizing: border-box;">VAL_DATA_ROOT</span>=examples/imagenet/myself/

<span class="hljs-constant" style="box-sizing: border-box;">RESIZE</span>=true</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li></ul>

RESIZE=true表示要在这里resize图像大小，如果预处理时没有做过resize这里一定要设置。GLOG_logtostderr=1是转储信息供人检查的，可以忽略它。运行：

<code class="language-shell hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">./examples/imagenet/myself/create_imagenet<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sh</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

在myself下生成lmdb格式的图像信息。

2.计算图像均值

模型要求从每个图像中减去图像均值，因此先要用tools/compute_image_mean.cpp计算均值，这个cpp同时也可以让我们熟练掌握如何控制多个组件，例如protocol buffer，leveldb，logging。均值可以运行如下命令计算：

<code class="language-shell hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">./examples/imagenet/myself/make_imagenet_mean<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sh</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

运行前修改一下几个路径：

<code class="hljs ini has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-setting" style="box-sizing: border-box;">EXAMPLE=<span class="hljs-value" style="box-sizing: border-box;">examples/imagenet/myself</span></span>
<span class="hljs-setting" style="box-sizing: border-box;">DATA=<span class="hljs-value" style="box-sizing: border-box;">examples/imagenet/myself</span></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

运行后在myself下生成imagenet_mean.binaryproto文件。

3.模型定义

下面介绍的算法应用是由Krizhevsky、Sutskever和Hinton在NIPS 2012 paper中提出的。网络定义models/bvlc_reference_caffenet/train_val.prototxt基本遵循了Krizhevsky的网络。我们前面自己设置的路径和这个prototxt中的不一样，那么我们把train_val.prototxt也复制到myself下，修改data层的路径。 
仔细看一下train_val.prototxt，注意到有好几个include部分特别指明了phase: TRAIN或者phase: TEST，这些部分让我们能够在同一个文件中定义用于训练的网络和用于测试的网络。这两个网络几乎一模一样，它们共享所有的层，除非指明了是include { phase: TRAIN }还是include { phase: TEST }，在这个例子中只有输入层和一个输出层是不同的。 
输入层的不同：训练网络的data输入层从examples/imagenet/myself/ilsvrc12_train_lmdb中取数据，并随机地对输入图像取镜像，测试网络的data层从examples/imagenet/myself/ilsvrc12_val_lmdb中取数据，且不执行随机镜像。 
输出层的不同：两种网络都输出softmax_loss层，它在训练时用于计算损失函数以及初始化反向传播，在测试时没什么用。测试网络同样有额外的输出层accuracy，用于报告测试集上的准确率，在训练过程中，偶尔会在测试集上测试，产生Test score #0: xxx和Test score #1: xxx这样的输出。其中，score 0是准确率（未训练的网络是1/5=0.2，大约会从这么多开始），score 1是损失（未训练的网络是7，大约会从这么多开始）。 
我们把models/bvlc_reference_caffenet/solver.prototxt也复制到myself下，把其中的网络模型路径也改成我们自己的，下面是具体参数设计（根据自己的数据集做了一点修改）： 
* 每一个batch有64（防止内存不够）个图像，总共是10,000次迭代 
* 每200次迭代，在验证数据上测试一次学到的网络 
* 设置初始化的学习率为0.003，每2,000次迭代后减少一次 
* 信息每20次迭代会显示一次 
* 网络训练的动量为0.9，权重衰减为0.00015 
* 对于每1,000次迭代，取一次当前状态的快照

<code class="language-makefile hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">net: "examples/imagenet/myself/model/train_val.prototxt"
test_iter: 100
test_interval: 200
base_lr: 0.003
lr_policy: "step"
gamma: 0.1
stepsize: 2000
display: 20
max_iter: 10000
momentum: 0.9
weight_decay: 0.00015
snapshot: 1000
snapshot_prefix: "examples/imagenet/myself/model/caffenet_train"
solver_mode: GPU</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li></ul>

4.训练ImageNet

把train_caffenet.sh也复制进myself，修改里面的内容为：

<code class="language-makefile hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">./build/tools/caffe train \
    --solver=examples/imagenet/myself/model/solver.prototxt</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

运行脚本：

<code class="language-shell hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">./examples/imagenet/myself/train_caffenet<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sh</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

非常非常慢，我训练完这些大约花了两个半小时，最后的准确率是82.4%。 
如果想要深度剖析计算时间，可以把下面的命令写到train_caffenet.sh中或直接运行：

<code class="language-shell hljs fix has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-attribute" style="box-sizing: border-box;">./build/tools/caffe time –model</span>=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">examples/imagenet/myself/model/train_val.prototxt</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

5.恢复训练

如果出了什么意外中断了训练，那真是天都要塌了，所以快照存储了训练的中间结果，这个设计真是人性化，当再次训练时，就可以从快照中恢复数据了。把resume_training.sh也复制进myself，修改内容为：

<code class="language-makefile hljs  has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">./build/tools/caffe train \
    --solver=examples/imagenet/myself/model/solver.prototxt \
    --snapshot=examples/imagenet/myself/model/caffenet_train_1000.solverstate</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li></ul>

这里的caffenet_train_1000.solverstate就是存储了恢复solver状态所需的所有必要信息的快照，具体是哪个solverstate是可以自己改的。 
运行脚本：

<code class="language-shell hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">./examples/imagenet/myself/resume_training<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.sh</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

得到训练好的模型后，下一步思考一下如何把它和Python接口结合起来使用classifying ImageNet