Convolutional Neural Network 卷积神经网络是基于人工神经网络提出的。人工神经网络模拟人的神经系统，由一定数量的神经元构成。在一个监督学习问题中，有一组训练数据 (xi,yi) ，x是样本，y是label，把它们输入人工神经网络，会得到一个非线性的分类超平面 hw,b(x) ，在这篇笔记中先梳理一下传统人工神经网络的基本概念，再基于传统人工神经网络简单介绍卷积神经网络。
1，神经元neuron
一个神经元是神经网络中的一个运算单元，它实质上就是一个函数。下图是一个神经元的示意图：
有3个输入
x1,x2,x3
，+1是一个偏置值(bias)，输出
hw,b(x)=f(wTx)=f(∑3i=1wixi+b)
，f是激活函数(activation function)，这里的激活函数用的是sigmoid函数：
f(z)=11+exp(−z).
sigmoid函数图像如图：
激活函数是以0.5为分界，让大值更逼近1，小值更逼近0，因为人的神经元就是有分“激活”和“不激活”两种状态，刺激达到一定程度，就被激活，传播信号，达不到就不激活，不传播信号，人工神经网络中激活函数的设计，应该是为了模拟这个过程。
2，人工神经网络Neural Network
人工神经网络就是由多个上述的神经元组合成的，下图是一个小型的人工神经网络的示意图：
有3个输入单元，这个输入单元的个数一般由样本向量的维度决定，比如输入的样本是sift特征点，那么就有128个输入单元。+1是偏置，
a(l)i
是指第l层的激活函数。这个神经网络的参数是
(W(1),b(2),W(2),b(2))
，其中
W(1)
是指第1层到第2层之间的连接的权值，是一个矩阵，
b(1)
是指它们之间的连接偏置，是一个向量。
W(l)i,j
指的是第l层的unit j和第l+1层的unit i之间的连接的权值，它是一个数值，
b(l)i
是连接到第l+1层unit i的偏置，所有连接unit i的连接都使用这同一个偏置值，在图中连接到第2层的三个unit的偏置值都是+1。在图中，L1是输入层(input layer)，L2是隐层(hidden layer)，因为这一层每个unit的值是无法观测到的，L3是输出层(output layer)。
当然输出层也可以有多个unit，如下图，在label y是向量的情况下，这个unit个数一般和y的维度一致。
3，卷积神经网络
卷积神经网络中，输入就是一幅幅的图像，权值W就是卷积模板，一般是卷积层和下采样层交替，最后是全连接的神经网络，也就是上述经典的人工神经网络。如下是一个简单的卷积神经网络示意图：
C是卷积层，S是下采样层。输入的一幅图像，在C1层，通过和3个卷积模板做卷积运算，然后加上偏置值，再经过sigmoid激活函数，得到3幅输出图像，在S2层，对C1层输出的3幅图像做下采样，假设采样因子是2，也就是图中每2*2的patch中的4个像素进行求和，再加偏置，再通过激活函数，得到3张尺寸减小了的输出图像。同样的，再经过C3,S4。将S4的输出拉成一个向量，输入传统的神经网络中，并得到输出。
如下图是有人对卷积神经网络每一层输出的特征图像做的可视化：
可以看到，第1层输出的特征图像还是很简单的形状和颜色信息，随着层数增多，到第3层的输出特征图，已经包含了相当复杂的丰富的细节信息，能够初步辨别目标了。
4，感受野(receptive field)和权值共享
除了增加卷积层和采样层，在传统人工神经网络的基础上，卷积神经网络还通过感受野和权值共享大大地减少了参数，降低了训练难度，也避免了参数过多引起过拟合(overfitting)。
卷积神经网络的输入是一幅幅的图像，根据传统的人工神经网络，在第2层的每个unit都要和输入层图像的每个像素连接，这里我们假设第2层有1M个hidden units，输入图像大小1000*1000，那么这一层要训练的权值w就是1M *1000*1000=10^12个，如下图左：
而人们发现人是通过一个局部的感受野去感受图像的，且图像的空间联系是局部的，每一个神经元并不需要对全局图像感受，每个神经元只用感受局部的图像区域，然后在更高层将这些神经元的信息综合起来得到全局信息。这样，我们就可以减少连接数目了，假设感受野是10*10大小，那么第2层的每个unit只用和一个10*10的感受野连接，这样的感受野有100*100个，那么这一层要训练的权值w是1M*10*10=10^8个，如上图右。可以看到，通过感受野，隐层的权值个数已经从10^12降到了10^8个，每个hidden unit有10*10=100个参数。
人们认为对于不同的感受野区域，要提取的特征是相似的，比如边缘，颜色变化等等，这样，我们可以在不同的感受野用同样的方式去感受特征，也就是说，用一个卷积核去卷积全部的感受野。这就是权值共享，就是让这1M个hidden units共享同一组参数(100个权值)，那么隐层权值个数就降到了100个。下图左中的红色，黑色，绿色，蓝色的感受野区域使用相同的参数来做卷积。
如果一种卷积核(也就是滤波器)可以提取图像的某一方面的特征，那么我们在这里使用100种卷积核，每个卷积核的参数都不一样，提取出来的特征也就不一样，这样，就可以输出100种特征图。上图右是红色和黑色两种卷积核，在卷积层会输出2中特征图。
Reference
http://ufldl.stanford.edu/wiki/index.php/Neural_Networks
http://blog.csdn.net/zouxy09/article/details/8781543
《Visualizing and Understanding Convolutional Networks》
在上篇笔记《CNN卷积神经网络学习笔记1：背景介绍》中已经介绍过CNN的结构，这篇笔记中，通过一个简单的CNN的例子，梳理一下CNN的网络结构的细节。
以下是一个6层的CNN网络，这个简单的CNN网络是DeepLearning Toolbox里面CNN的一个例子，后面要通过DeepLearning Toolbox中CNN的代码来进一步理解CNN的整个过程。我们输入的是1张大小为28*28的图片。
需要注意的有：
1，这里输入的是一张图片，如果我们输入了50张图片，那么下图中的每一个方框（代表一种特征图）实际上代表了50张图片。
2，在S3和S5的采样层只做了mean pooling，其实一般还会有加偏置和激活的操作，这个CNN网络比较简单，省略了这两步。
3，C4卷积层是用每个卷积核对所有的每一种输入特征图做卷积，再求和得到一种输出特征图，但一般是选择几种输入特征图来做卷积，并不是全部。
在上篇《CNN卷积神经网络学习笔记2：网络结构》中，已经介绍了CNN的网络结构的详细构成，我们已经可以初始化一个自己的CNN网络了，接下来就是要用训练得到一个确定的CNN的模型，也就是确定CNN的参数。
CNN本质上就是人工神经网络的一种，只是在前几层的处理上有所不同，我们可以把卷积核看成是人工神经网络里的权值W，而采样层实质上也是一种卷积运算。所以可以基于人工神经网络的权值更新的方法来推导CNN里的权值更新公式。人工神经网络里是用反向传播算法将误差层层回传，利用梯度下降法更新每一层的权值，CNN中也是类似的。所以这里先对传统的BP算法做个概述，然后再推广到CNN中。
1，BP算法
1.1 Feedforward Pass前向传播
首先定义平方误差代价函数：
EN=12∑n=1N∑k=1c(tnk−ynk)2.
其中N是样本个数，c是label的维度，对于分类问题，意味着这些样本能分为c类。 tkn 表示第n个样本的label tn 的第k维， ykn 是第n个样本网络的输出(predict label)的第k维。我们的目标是要更新网络的权值，使得网络输出y与真实值t更接近，也就是最小化这个E，考虑到要考虑一个样本，则第n个样本的误差函数是：
En=12∑k=1c(tnk−ynk)2.
定义第l层的输出为：
xl=f(ul),其中ul=Wlxl−1+bl(1)
这里f是激活函数， xl−1 是l-1层的输出，也就是l层的输入，W和b分别是l层的权值和偏置。
上式就是前向传播的公式，每一层对输入(也就是上一层的输出)做运算，得到输出结果，这样将样本信息逐层传递，最后输出一个预测值(predict label)。
1.2 Backpropagation Pass反向传播
预测值与真实值(label)之间必然是存在误差的，反向传播就是要把这个误差信息回传给每一层，让这些层修改他们的权值，使得CNN更精准。
BP算法中是用梯度下降法更新权值的，梯度下降法的更新公式如下：
Wlnew=Wlold−η∂E∂Wlold.
blnew=blold−η∂E∂blold.
梯度下降法细节可以参考这里：
http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization
其中 η 是梯度下降的学习率(learning rate)，可以看出，梯度下降法更新权值主要是利用误差代价函数对参数的梯度，所以权值更新的目标就是让每一层得到这样的梯度，然后更新。
为了求取单个样本的误差代价函数对参数的偏导，这里定义节点灵敏度(sensitivities) δ 为误差对输出的变化率：
δ=∂E∂u
其中的u是 ul=Wlxl−1+bl .
对于参数中的偏置b，因为 ∂u∂b=1 ，由链式求导法则可得：
∂E∂bl=∂E∂ul∂ul∂bl=δl.(2)
每层的灵敏度是不一样的，可以算得：
δl=∂E∂bl=∂12(y−t)2∂bl=f′(ul)∘(yn−tn).(3)
注意这里y也是b的函数， y=f(ul)=f(Wlxl−1+b) ，所以要乘上 f′(ul) ，这里的 ∘ 表示每个元素相乘，因为每个神经元连接都会有一个灵敏度 δ ，所以每一层的灵敏度是一个矩阵。
进一步求得误差代价函数E对参数中的权值W的偏导：
∂E∂Wl=∂E∂ul∂ul∂Wl=δlxl−1.(4)
至此，我们得到了每一层利用梯度下降进行权值更新时需要的梯度，也就是(2),(4)，可以看到他们都和灵敏度有关，而灵敏度可由(3)式计算。
在(3)式中， yl 和 ul 中的 xl−1 是不知道的，也就是说，我们不知道每一层具体的的输入和输出，而且这个也太难计算，那么怎么把误差信息层层回传呢？
从灵敏度下手：
δl=∂E∂ul=∂E∂ul+1∂ul+1∂ul=δl+1∂(Wl+1xl+b)∂ul=δl+1∂(Wl+1f(ul)+b)∂ul=δl+1Wl+1∘f′(ul).
所以反向传播其实是通过灵敏度层层回传误差信息，如下就是反向传播的核心公式：
δl=δl+1Wl+1∘f′(ul).(5)
以上是对经典的BP算法做一个概述，CNN中把权值W换成卷积核k，按照(1)(2)(3)(4)(5)式就可以得到CNN的权值更新公式。
2，CNN中卷积层权值更新推导
2.1 对照(1)式计算l层的输出
在CNN中，对于卷积层的每一种输出的特征图
xj
有：
xlj=f(∑i∈Mjxl−1i∗klij+bj).
其中，Mj表示选择的输入特征图组合， kij 是输入的第i种特征图和输出的第j种特征图之间的连接所用的卷积核， bj 是第j种特征图对应的偏置，f是激活函数。
2.2 对照(5)式计算灵敏度
δlj=δl+1jWl+1j∘f′(ul)=βl+1jup(δl+1j)∘f′(ul).
因为l+1层是采样层，所以相当于也是做卷积，例如做scale=2的下采样，就是用2*2的每个值为1/4的卷积核卷积图像，所以这里的权值W实际上就是这个2*2的卷积核，它的值是 βj 。up表示上采样操作，因为l+1采样层的灵敏度矩阵是l层灵敏度矩阵的尺寸的1/4(scale=2时)，所以这里要对l+1层的灵敏度矩阵做上采样，使它们尺寸一致。
2.3 对照(2)式计算误差代价函数对偏置b的偏导
也就是对层l中的灵敏度中所有节点求和，这里(u,v)代表灵敏度矩阵中的元素位置：
∂E∂bj=∑u,v(δlj)u,v
2.4 对照(4)式计算误差代价函数对卷积核k的偏导：
∂E∂klij=∑u,v(δlj)u,v(pl−1i)uv.
这里 (pl−1i)uv 是 xl−1i 在做卷积时，与 kij 做卷积的每一个patch，(u,v)是patch中心，输出特征图中(u,v)位置的值，是由输入特征图中(u,v)位置的patch和卷积核 kij 卷积所得的值。
3，CNN中下采样层权值更新推导
3.1 对照(1)式计算l层的输出
在CNN中，对于采样层的每一种输出特征图
xj
有：
xlj=f(βljdown(xl−1j)+blj).
down表示下采样，这里的 β 是乘性偏置，b是加性偏置，一般cnn网络中没有这个 β 。
3.2 对照(5)式计算灵敏度
δlj=δl+1jWl+1j∘f′(ul)=f′(ulj)∘conv2(δl+1j,rot180(kl+1j),′full′).
3.3 对照(2)式计算误差代价函数对偏置b的偏导
这里和卷积层的b是一样的：
∂E∂bj=∑u,v(δlj)u,v.
至此我们就得到了CNN的权值更新公式。
下一篇中讨论一个简单的CNN实现。
Reference
《Notes on Convolutional Neural Networks》
http://cogprints.org/5869/1/cnn_tutorial.pdf
以及它的中文翻译：
http://blog.csdn.net/zouxy09/article/details/9993371
代码来自github上的一个DeepLearning Toolbox，地址：https://github.com/rasmusbergpalm/DeepLearnToolbox
主要参考这篇博客中的代码注释：
http://blog.csdn.net/zouxy09/article/details/9993743
单步调试后自己添加了一部分注释
如下：
test_example_CNN
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">function test_example_CNN load mnist_uint8; train_x = double(reshape(train_x<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">',28,28,60000))/255; test_x = double(reshape(test_x'</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">28</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">28</span>,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10000</span>))/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">255</span>; train_y = double(train_y<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'); test_y = double(test_y'</span>); <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %</span>will run <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> epoch <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> about <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">200</span> second <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> get around <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span>% error. %<span class="hljs-constant" style="box-sizing: border-box;">With</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span> epochs you<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'ll get around 1.2% error rand('</span>state<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">',0) cnn.layers = { struct('</span>type<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', '</span>i<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">') %input layer struct('</span>type<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', '</span>c<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', '</span>outputmaps<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', 6, '</span>kernelsize<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', 5) %convolution layer struct('</span>type<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', '</span>s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', '</span>scale<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', 2) %sub sampling layer struct('</span>type<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', '</span>c<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', '</span>outputmaps<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', 12, '</span>kernelsize<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', 5) %convolution layer struct('</span>type<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', '</span>s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', '</span>scale<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">', 2) %subsampling layer }; % learning rate 学习率，决定收敛的速度 opts.alpha = 1; % 把样本split为每50个一份，每用batchsize个样本训练一次，调整一次权值 opts.batchsize = 50; % split次数 opts.numepochs = 1; cnn = cnnsetup(cnn, train_x, train_y); cnn = cnntrain(cnn, train_x, train_y, opts); [er, bad] = cnntest(cnn, test_x, test_y); %plot mean squared error figure; plot(cnn.rL); assert(er<0.12, '</span><span class="hljs-constant" style="box-sizing: border-box;">Too</span> big error<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'); </span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li></ul>
cnnsetup
<code class="hljs matlab has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">function</span> <span class="hljs-title" style="box-sizing: border-box;">net</span> = <span class="hljs-title" style="box-sizing: border-box;">cnnsetup</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(net, x, y)</span></span> inputmaps = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% B=squeeze(A) 返回和矩阵A相同元素但所有单一维都移除的矩阵B，单一维是满足size(A,dim)=1的维。</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% train_x中图像的存放方式是三维的reshape(train_x',28,28,60000)，前面两维表示图像的行与列，</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 第三维就表示有多少个图像。这样squeeze(x(:, :, 1))就相当于取第一个图像样本后，再把第三维</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 移除，就变成了28x28的矩阵，也就是得到一幅图像，再通过size函数就得到了训练样本图像的行数与列数</span> mapsize = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">size</span>(<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">squeeze</span>(x(:, :, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 下面通过传入net这个结构体来逐层构建CNN网络</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% n = numel(A)返回数组A中元素个数</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% net.layers中有五个struct类型的元素，实际上就表示CNN共有五层，这里的范围是5</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> l = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">numel</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% layer</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> strcmp(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>type, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'s'</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 如果这层是 子采样层</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% subsampling层的mapsize，最开始mapsize是每张图的大小28*28</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 这里除以scale=2，就是pooling之后图的大小，pooling域之间没有重叠，所以pooling后的图像为14*14</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 注意这里的右边的mapsize保存的都是上一层每张特征map的大小，它会随着循环进行不断更新</span> mapsize = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">floor</span>(mapsize / <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>scale); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">j</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : inputmaps <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% inputmap就是上一层有多少张特征图</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>b<span class="hljs-cell" style="box-sizing: border-box;">{j}</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 将偏置初始化为0</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> strcmp(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>type, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'c'</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 如果这层是 卷积层</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 旧的mapsize保存的是上一层的特征map的大小，那么如果卷积核的移动步长是1，那用</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% kernelsize*kernelsize大小的卷积核卷积上一层的特征map后，得到的新的map的大小就是下面这样</span> mapsize = mapsize - <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>kernelsize + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 该层需要学习的参数个数。每张特征map是一个(后层特征图数量)*(用来卷积的patch图的大小)</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 因为是通过用一个卷积核在上一个特征map层中移动（核窗口每次移动1个像素），遍历上一个特征map</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 层的每个神经元。卷积核由kernelsize*kernelsize个元素组成，每个元素是一个独立的权值，所以</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 就有kernelsize*kernelsize个需要学习的权值，再加一个偏置值。另外，由于是权值共享，也就是</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 说同一个特征map层是用同一个具有相同权值元素的kernelsize*kernelsize的核窗口去感受输入上一</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 个特征map层的每个神经元得到的，所以同一个特征map，它的权值是一样的，共享的，权值只取决于</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 核窗口。然后，不同的特征map提取输入上一个特征map层不同的特征，所以采用的核窗口不一样，也</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 就是权值不一样，所以outputmaps个特征map就有（kernelsize*kernelsize+1）* outputmaps那么多的权值了</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 但这里fan_out只保存卷积核的权值W，偏置b在下面独立保存</span> fan_out = <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>outputmaps * <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>kernelsize ^ <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">j</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>outputmaps <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% output map</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% fan_out保存的是对于上一层的一张特征map，我在这一层需要对这一张特征map提取outputmaps种特征，</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 提取每种特征用到的卷积核不同，所以fan_out保存的是这一层输出新的特征需要学习的参数个数</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 而，fan_in保存的是，我在这一层，要连接到上一层中所有的特征map，然后用fan_out保存的提取特征</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 的权值来提取他们的特征。也即是对于每一个当前层特征图，有多少个参数链到前层</span> fan_in = inputmaps * <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>kernelsize ^ <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">i</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : inputmaps <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% input map</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 随机初始化权值，也就是共有outputmaps个卷积核，对上层的每个特征map，都需要用这么多个卷积核</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 去卷积提取特征。</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% rand(n)是产生n×n的 0-1之间均匀取值的数值的矩阵，再减去0.5就相当于产生-0.5到0.5之间的随机数</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 再 *2 就放大到 [-1, 1].</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 反正就是将卷积核每个元素初始化为[-sqrt(6 / (fan_in + fan_out)), sqrt(6 / (fan_in + fan_out))]</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 之间的随机数。因为这里是权值共享的，也就是对于一张特征map，所有感受野位置的卷积核都是一样的</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 所以只需要保存的是 inputmaps * outputmaps 个卷积核。</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>k<span class="hljs-cell" style="box-sizing: border-box;">{i}</span><span class="hljs-cell" style="box-sizing: border-box;">{j}</span> = (<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">rand</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>kernelsize) - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.5</span>) * <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span> * <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">sqrt</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span> / (fan_in + fan_out)); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>b<span class="hljs-cell" style="box-sizing: border-box;">{j}</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 将偏置初始化为0</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 只有在卷积层的时候才会改变特征map的个数，pooling的时候不会改变个数。这层输出的特征map个数就是</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 输入到下一层的特征map个数</span> inputmaps = <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>outputmaps; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% fvnum 是输出层的前面一层的神经元个数。</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 这一层的上一层是经过pooling后的层，包含有inputmaps个特征map。每个特征map的大小是mapsize。</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 所以，该层的神经元个数是 inputmaps * （每个特征map的大小）</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% prod: Product of elements.</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% For vectors, prod(X) is the product of the elements of X</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 在这里 mapsize = [特征map的行数 特征map的列数]，所以prod后就是 特征map的行*列</span> fvnum = prod(mapsize) * inputmaps; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% onum 是标签的个数，也就是输出层神经元的个数。要分多少个，就有多少个输出神经元</span> onum = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">size</span>(y, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 这里是最后一层神经网络的设定</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% ffb 是输出层每个神经元对应的基biases</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>ffb = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">zeros</span>(onum, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% ffW 输出层前一层 与 输出层 连接的权值，这两层之间是全连接的</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>ffW = (<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">rand</span>(onum, fvnum) - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.5</span>) * <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span> * <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">sqrt</span>(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span> / (onum + fvnum)); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li></ul>
cnntrain
<code class="hljs matlab has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">function</span> <span class="hljs-title" style="box-sizing: border-box;">net</span> = <span class="hljs-title" style="box-sizing: border-box;">cnntrain</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(net, x, y, opts)</span></span> m = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">size</span>(x, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% m 保存的是 训练样本个数</span> numbatches = m / <span class="hljs-transposed_variable" style="box-sizing: border-box;">opts.</span>batchsize; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% rem: Remainder after division. rem(x,y) is x - n.*y 相当于求余</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% rem(numbatches, 1) 就相当于取其小数部分，如果为0，就是整数</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">rem</span>(numbatches, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>) ~= <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> error(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'numbatches not integer'</span>); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>rL = <span class="hljs-matrix" style="box-sizing: border-box;">[]</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">i</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-transposed_variable" style="box-sizing: border-box;">opts.</span>numepochs <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% disp(X) 打印数组元素。如果X是个字符串，那就打印这个字符串</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">disp</span>(<span class="hljs-matrix" style="box-sizing: border-box;">[<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'epoch '</span> num2str(i) <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'/'</span> num2str(opts.numepochs)]</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% tic 和 toc 是用来计时的，计算这两条语句之间所耗的时间</span> tic; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% P = randperm(N) 返回[1, N]之间所有整数的一个随机的序列，例如</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% randperm(6) 可能会返回 [2 4 5 6 1 3]</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 这样就相当于把原来的样本排列打乱，再挑出一些样本来训练</span> kk = randperm(m); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> l = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : numbatches <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 取出打乱顺序后的batchsize个样本和对应的标签</span> batch_x = x(:, :, kk((l - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>) * <span class="hljs-transposed_variable" style="box-sizing: border-box;">opts.</span>batchsize + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : l * <span class="hljs-transposed_variable" style="box-sizing: border-box;">opts.</span>batchsize)); batch_y = y(:, kk((l - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>) * <span class="hljs-transposed_variable" style="box-sizing: border-box;">opts.</span>batchsize + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : l * <span class="hljs-transposed_variable" style="box-sizing: border-box;">opts.</span>batchsize)); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 在当前的网络权值和网络输入下计算网络的输出</span> net = cnnff(net, batch_x); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% Feedforward</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 得到上面的网络输出后，通过对应的样本标签用bp算法来得到误差对网络权值</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">%（也就是那些卷积核的元素）的导数</span> net = cnnbp(net, batch_y); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% Backpropagation</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 得到误差对权值的导数后，就通过权值更新方法去更新权值</span> net = cnnapplygrads(net, opts); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">isempty</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>rL) <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>rL(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>) = <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>L; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 代价函数值，也就是误差值</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>rL(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>) = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.99</span> * <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>rL(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span>) + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.01</span> * <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>L; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 保存历史的误差值，以便画图分析</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> toc; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li></ul>
cnnff
<code class="hljs matlab has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">function</span> <span class="hljs-title" style="box-sizing: border-box;">net</span> = <span class="hljs-title" style="box-sizing: border-box;">cnnff</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(net, x)</span></span> n = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">numel</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 层数</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.</span>a<span class="hljs-cell" style="box-sizing: border-box;">{<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}</span> = x; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 网络的第一层就是输入，但这里的输入包含了多个训练图像</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 输入层只有一个特征图，也就是原始的输入图像，注意这里所说的特征图，并不是指一张图片，而是指一类图片，这一类图片是包含同种特征信息的。</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 即使这里是50张图片，也算作是一个特征图。 </span> inputmaps = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> l = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span> : n <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% for each layer</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> strcmp(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>type, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'c'</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 卷积层</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% !!below can probably be handled by insane matrix operations</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 对每一个输入map，或者说我们需要用outputmaps个不同的卷积核去卷积图像</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">j</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>outputmaps <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% for each output map</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% create temp output map</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 对上一层的每一张特征map，卷积后的特征map的大小就是 </span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% （输入map宽 - 卷积核的宽 + 1）* （输入map高 - 卷积核高 + 1）</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 对于这里的层，因为每层都包含多张特征map，对应的索引保存在每层map的第三维</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 所以，这里的z保存的就是该层中所有的特征map了</span> z = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">zeros</span>(<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">size</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.</span>a<span class="hljs-cell" style="box-sizing: border-box;">{<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}</span>) - <span class="hljs-matrix" style="box-sizing: border-box;">[net.layers{l}.kernelsize - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> net.layers{l}.kernelsize - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>]</span>); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">i</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : inputmaps <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% for each input map</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% convolve with corresponding kernel and add to temp output map</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 用当前层的每一个卷积核去卷积输入的所有的inputmaps张特征图，然后把这些卷积结果加起来，得到一张输出特征图</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 输出特征图的个数就是卷积核个数，也就是outputmaps， </span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 一个特征图是指含有同一类特征的图集合，我们对这一类图片用同一种卷积核做卷积，提取出同一种特征。</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 另外，有些论文或者实际应用中，并不是与全部的特征map链接的，有可能只与其中的某几个连接</span> z = z + convn(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.</span>a<span class="hljs-cell" style="box-sizing: border-box;">{i}</span>, <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>k<span class="hljs-cell" style="box-sizing: border-box;">{i}</span><span class="hljs-cell" style="box-sizing: border-box;">{j}</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'valid'</span>); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% add bias, pass through nonlinearity</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 加上对应位置的基b，然后再用sigmoid函数算出特征图中每个位置的激活值，作为该层输出特征图</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>a<span class="hljs-cell" style="box-sizing: border-box;">{j}</span> = sigm(z + <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>b<span class="hljs-cell" style="box-sizing: border-box;">{j}</span>); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% set number of input maps to this layers number of outputmaps</span> inputmaps = <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>outputmaps; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">elseif</span> strcmp(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>type, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'s'</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 下采样层</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% downsample</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">j</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : inputmaps <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% !! replace with variable</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 例如我们要在scale=2的域上面执行mean pooling，那么可以卷积大小为2*2，每个元素都是1/4的卷积核</span> z = convn(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.</span>a<span class="hljs-cell" style="box-sizing: border-box;">{j}</span>, <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">ones</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>scale) / (<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>scale ^ <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>), <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'valid'</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 因为convn函数的默认卷积步长为1，而pooling操作的域是没有重叠的，所以对于上面的卷积结果</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 最终pooling的结果需要从上面得到的卷积结果中以scale=2为步长，跳着把mean pooling的值读出来</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>a<span class="hljs-cell" style="box-sizing: border-box;">{j}</span> = z(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>scale : <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>scale : <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span>, :); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% concatenate all end layer feature maps into vector</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 把最后一层得到的特征图拉成一条向量，作为最终提取到的特征向量</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>fv = <span class="hljs-matrix" style="box-sizing: border-box;">[]</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">j</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">numel</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{n}.</span>a) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 最后一层的特征map的个数</span> sa = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">size</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{n}.</span>a<span class="hljs-cell" style="box-sizing: border-box;">{j}</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 第j个特征map的大小</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 将所有的特征图拉成一条列向量。还有一维就是对应的样本索引。每个样本一列，每列为对应的特征向量</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>fv = <span class="hljs-matrix" style="box-sizing: border-box;">[net.fv; reshape(net.layers{n}.a{j}, sa(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>) * sa(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>), sa(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>))]</span>; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% feedforward into output perceptrons</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 计算网络的最终输出值。sigmoid(W*X + b)，注意是同时计算了batchsize个样本的输出值</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>o = sigm(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>ffW * <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>fv + <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">repmat</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>ffb, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>, <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">size</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>fv, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>))); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li></ul>
cnnbp
<code class="hljs matlab has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">function</span> <span class="hljs-title" style="box-sizing: border-box;">net</span> = <span class="hljs-title" style="box-sizing: border-box;">cnnbp</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(net, y)</span></span> n = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">numel</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 网络层数</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% error</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% net.o 是label的预测值，y是真实值</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>e = <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>o - y; <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% loss function</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 代价函数是 均方误差</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>L = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>* sum(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>e(:) .^ <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>) / <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">size</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>e, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">%% backprop deltas</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 这里可以参考 UFLDL 的 反向传导算法 的说明</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 输出层的 灵敏度 或者 残差</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 这里对应的就是前文公式推导的灵敏度计算公式 sigma^l = (y - t)* f'(u^l), 这个f是sigmoid函数</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% net.e 是 (y - t)；(net.o .* (1 - net.o))是f'(u).</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>od = <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>e .* (<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>o .* (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> - <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>o)); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% output delta</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 灵敏度 反向传播回 前一层</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 对应公式：sigma^l = (W^(l+1))' * sigma^(l+1) * f'(u^l)</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% net.ffW'是(W^(l+1))'，因为最后一层是全连接层；sigma^(l+1)是这里的net.od</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 这里的下采样层没有sigmoid函数，只有卷积层有</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>fvd = (<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span><span class="hljs-transposed_variable" style="box-sizing: border-box;">ffW'</span> * <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>od); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% feature vector delta</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> strcmp(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{n}.</span>type, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'c'</span>) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% only conv layers has sigmoid function</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>fvd = <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>fvd .* (<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>fv .* (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> - <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>fv)); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% reshape feature vector deltas into output map style</span> sa = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">size</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{n}.</span>a<span class="hljs-cell" style="box-sizing: border-box;">{<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 最后一层特征map的大小。这里的最后一层都是指输出层的前一层</span> fvnum = sa(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>) * sa(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 因为是将最后一层特征map拉成一条向量，所以对于一个样本来说，特征维数是这样</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">j</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">numel</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{n}.</span>a) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 最后一层的特征map的组数</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 在fvd里面保存的是所有样本的特征向量（在cnnff.m函数中用特征map拉成的），所以这里需要重新</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 变换回来矩阵的形式。d 保存的是 delta，也就是 灵敏度</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{n}.</span>d<span class="hljs-cell" style="box-sizing: border-box;">{j}</span> = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">reshape</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>fvd(((<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">j</span> - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>) * fvnum + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>) : <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">j</span> * fvnum, :), sa(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>), sa(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>), sa(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>)); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 对于 输出层前面的层（与输出层计算灵敏度的方式不同）</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> l = (n - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>) : -<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> strcmp(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>type, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'c'</span>) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">j</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">numel</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>a) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 该层特征map的组数</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% net.layers{l}.d{j} 保存的是 第l层 的 第j个 map 的 灵敏度map。 也就是每个神经元节点的delta的值</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% expand的操作相当于对l+1层的灵敏度map进行上采样。然后前面的操作相当于对该层的输入a进行sigmoid求导</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 这里对应的公式是：sigma^l(j) = f'(u^l(j)) * up(sigma^(l+1)(j))</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% net.layers{l}.a{j} .* (1 - net.layers{l}.a{j})就是f'(u^l(j))</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 后面的expand就是up</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 前文的公式推到中，还多乘了一个beta，这个beta是下采样时map的权值，此处没有设置这个权值。</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>d<span class="hljs-cell" style="box-sizing: border-box;">{j}</span> = <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>a<span class="hljs-cell" style="box-sizing: border-box;">{j}</span> .* (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> - <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>a<span class="hljs-cell" style="box-sizing: border-box;">{j}</span>) .* (expand(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.</span>d<span class="hljs-cell" style="box-sizing: border-box;">{j}</span>, <span class="hljs-matrix" style="box-sizing: border-box;">[net.layers{l + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.scale net.layers{l + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.scale <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>]</span>) / <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.</span>scale ^ <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">elseif</span> strcmp(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>type, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'s'</span>) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">i</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">numel</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>a) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 第l层特征map的个数</span> z = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">zeros</span>(<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">size</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>a<span class="hljs-cell" style="box-sizing: border-box;">{<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}</span>)); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">j</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">numel</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.</span>a) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 第l+1层特征map的个数</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 这里特征map组合就是全部相加</span> z = z + convn(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.</span>d<span class="hljs-cell" style="box-sizing: border-box;">{j}</span>, rot180(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l + <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.</span>k<span class="hljs-cell" style="box-sizing: border-box;">{i}</span><span class="hljs-cell" style="box-sizing: border-box;">{j}</span>), <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'full'</span>); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>d<span class="hljs-cell" style="box-sizing: border-box;">{i}</span> = z; <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">%% calc gradients</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 这里与 Notes on Convolutional Neural Networks 中不同，这里的 子采样 层没有参数，也没有</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 激活函数，所以在子采样层是没有需要求解的参数的</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> l = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span> : n <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> strcmp(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>type, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'c'</span>) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">j</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">numel</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>a) <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">i</span> = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> : <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">numel</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.</span>a) <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% dk 保存的是 误差对卷积核 的导数</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 如果l=2，那么每个a{i}是一种特征图，是28*28*50，d{j}是灵敏度矩阵，大小是24*24*50，这里50是指输入了50张图</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% flipall是将每个28*28旋转180度，并且50张图的前后顺序会颠倒，逆序。</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% convn是将每个28*28和24*24做卷积，得到的结果大小是5*5，就是卷积核k的大小，然后把50个5*5的矩阵对应位置相加，所以后面这里要除以50，也就是size</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 也就是说这里计算的梯度是对50个图的平均</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>dk<span class="hljs-cell" style="box-sizing: border-box;">{i}</span><span class="hljs-cell" style="box-sizing: border-box;">{j}</span> = convn(flipall(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l - <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>}.</span>a<span class="hljs-cell" style="box-sizing: border-box;">{i}</span>), <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>d<span class="hljs-cell" style="box-sizing: border-box;">{j}</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'valid'</span>) / <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">size</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>d<span class="hljs-cell" style="box-sizing: border-box;">{j}</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% db 保存的是 误差对于bias基 的导数，也是对50个图的平均</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>db<span class="hljs-cell" style="box-sizing: border-box;">{j}</span> = sum(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>d<span class="hljs-cell" style="box-sizing: border-box;">{j}</span>(:)) / <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">size</span>(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>layers<span class="hljs-cell" style="box-sizing: border-box;">{l}.</span>d<span class="hljs-cell" style="box-sizing: border-box;">{j}</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>); <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 计算全连接层的参数的梯度</span> <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>dffW = <span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>od * (<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>fv)<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">' / size(net.od, 2); net.dffb = mean(net.od, 2); function X = rot180(X) X = flipdim(flipdim(X, 1), 2); end end </span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li></ul>
cnntest
<code class="hljs matlab has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-function" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">function</span> <span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">[er, bad]</span> = <span class="hljs-title" style="box-sizing: border-box;">cnntest</span><span class="hljs-params" style="color: rgb(102, 0, 102); box-sizing: border-box;">(net, x, y)</span></span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% feedforward</span> net = cnnff(net, x); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 前向传播得到输出</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% [Y,I] = max(X) returns the indices of the maximum values in vector I</span> <span class="hljs-matrix" style="box-sizing: border-box;">[~, h]</span> = max(<span class="hljs-transposed_variable" style="box-sizing: border-box;">net.</span>o); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 找到最大的输出对应的标签</span> <span class="hljs-matrix" style="box-sizing: border-box;">[~, a]</span> = max(y); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 找到最大的期望输出对应的索引</span> bad = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">find</span>(h ~= a); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 找到他们不相同的个数，也就是错误的次数</span> er = <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">numel</span>(bad) / <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">size</span>(y, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>); <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">% 计算错误率</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">end</span></code>
25万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
