Cost Function and Backpropagation

Cost Function and Backpropagation

Cost Function

Neural Network(Classification)

有m组训练集{$(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),...,(x^{(m)},y^{(m)})$}

$L$= total number of layers in network

$s_l$= number of units (not counting bias unit) in layer $l$

$\underline{Binaryclassification}$

y=0 or 1

1 output unit

$\underline{Multi-classclassification}$(K classes)

$y\in {\mathbb{R}}^K$

K output units

Cost function

​ Logistic regression:

$J(\theta )=-\frac{1}{m}{\sum }_{i=1}^m[y^{(i)}log(h_{\theta }(x^{(i)}))+(1-y^{(i)})log(1-h_{\theta }(x^{(i)}))]+\frac{\lambda }{2m}{\sum }_{j=1}^n{\theta }_j^2$

​ Neural network:

$h_{\Theta }(x)\in {\mathbb{R}}^K$ $(h_{\Theta }(x){)}_i=i^{th}$ output

$J(\Theta )=-\frac{1}{m}[{\sum }_{i=1}^m{\sum }_{k=1}^K{y}_k^{(i)}log(h_{\Theta }(x^{(i)}){)}_k+(1-y_k^{(i)})log(1-h_{\Theta }(x^{(i)}){)}_k]+\frac{\lambda }{2m}{\sum }_{l=1}^{L-1}{\sum }_{i=1}^{s_l}{\sum }_{j=1}^{s_{l+1}}({\Theta }_{ji}^{(l)}{)}^2$

Note:

• the double sum simply adds up the logistic regression costs calculated for each cell in the output layer
• the triple sum simply adds up the squares of all the individual Θs in the entire network.
• the i in the triple sum does not refer to training example i

Gradient computation: Backpropagation algorithm

反向传播算法

${\delta }_j^{(l)}$= “error” of node $j$ in layer $l$

Backpropagation algorithm

Training set {$(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),...,(x^{(m)},y^{(m)})$}

Set ${\Delta }_{ij}^{(l)}=0$(for all l,i,j) （用来作累加项计算偏导数）

For $i=1$ to m $(x^{(i)},y^{(i)})$

​ Set $a^{(1)}=x^{(i)}$

​ Perform forward propagation to compute $a^{(l)}$ for $l=2,3,...,L$

​ Using $y^{(i)}$, compute ${\delta }^{(L)}=a^{(L)}-y^{(i)}$

​ Compute ${\delta }^{(L-1)},{\delta }^{(L-2)},...,{\delta }^{(2)}$ using ${\delta }^{(l)}=(({\Theta }^{(l)}{)}^T{\delta }^{(l+1)}).\ast a^{(l)}.\ast (1-a^{(l)})$

​ ${\Delta }_{ij}^{(l)}:={\Delta }_{ij}^{(l)}+a_j^{(l)}{\delta }_i^{(l+1)}$(with vectorization, ${\Delta }^{(l)}:={\Delta }^{(l)}+{\delta }^{(l+1)}(a^{(l)}{)}^T$)

Hence update new $\Delta$ matrix:

• $D_{i,j}^{(l)}:=\frac{1}{m}\left({\Delta }_{i,j}^{(l)}+\lambda {\Theta }_{i,j}^{(l)}\right),ifj\ne 0$.
• $D_{i,j}^{(l)}:=\frac{1}{m}{\Delta }_{i,j}^{(l)}Ifj=0$

get $\frac{\partial }{\partial {\Theta }_{ij}^{(l)}}J(\Theta )=D_{ij}^{(l)}$

Backpropagation Intuition

固定步骤：

forward:

第一层$(x^{(i)},y^{(i)})$, 传播到第一个隐藏层时，计算出输入单元的加权和$z^{(2)}$

，通过sigmoid逻辑函数和sigmoid激活函数算得激活值$a^{(2)}$，继续同样的方法进行前向传播

第一个隐藏层输入到第二个隐藏层的权重为${\Theta }^{(2)}$,，可以得到关系式：

第三层$z^{(3)}={\Theta }_{10}^{(2)}\times 1+{\Theta }_{11}^{(2)}\times a_1^{(2)}+...$

反向传播则相反

代价函数表达式中：

$J(\theta )=-\frac{1}{m}{\sum }_{i=1}^m[y_k^{(t)}log(h_{\theta }(x^{(t)}){)}_k+(1-y_k^{(t)})log(1-h_{\theta }(x^{(t)}{)}_k)]+\frac{\lambda }{2m}{\sum }_{l=1}^{L-1}{\sum }_{i=1}^{sl}{\sum }_{j=1}^{sl+1}{\theta }_{j,i}^2$

$cost(t)=y(t)log(h_{\Theta }(x^{(t)}))+(1-y(t))log(1-h_{\Theta }(x^{(t)}))$可以看作一种方差函数 ($cost(t)\approx (h_{\Theta }(x^{(t)})-y^{(t)}{)}^2$)

可以直观地看到：

${\delta }_j^{(l)}=\frac{\partial }{\partial z_j^{(l)}}cost(t)$(for $j\ge 0$)

${\delta }_j^{(l)}$就是第l层第i个单元得到的激活项误差error

计算：从最后一层，假设为${\delta }_1^{(4)}=a^{(4)}-y$（预测值减去实际值）

继续向前传播算得${\delta }^{(3)}$等，计算过程：

${\delta }^{(3)}$=${\delta }^{(4)}\times {\Theta }_3$

Notice:All the $\delta$ values only for the hidden units but excluding the biased units.