人工神经网络 backpropagation algorithm

ann人工神经网络bp算法python实现

1、单个神经元

1.1、单个神经元的表示方法：

这代表一行数据的处理，即 $x_1,x_2,x_3$属于一个样本的三个维度，输出为这个样本的激活函数的值。

$$\textstyle h_{W,b}(x) = f(z)=f(W^Tx) = f(\sum_{i=1}^3 W_{i}x_i +b)$$

1.2、激励函数几种不同形式：

• sigmoid函数
  $$f(z) = \frac{1}{1+\exp(-z)}$$
  
  • sigmoid函数的导数：
    $$f'(z) = f(z) (1-f(z))$$
• tanh函数
  $$f(z) = \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$$
  
  • tanh函数的导数：
    $$f'(z) = 1- (f(z))^2$$

• rectified linear activation function
  

  $$f(z) = \max(0,x)$$
  • 导数为：
  $$f'(z) =\left\{\begin{matrix} 0 & z<=0 \\ 1 & z>0 \end{matrix}\right.$$
  以下是这几种损失函数的函数图。
  
  

2、神经网络模型

令 $n_l$表示这个神经网络的层数，这个神经网络共有3层，及$n_l=3$。
$L_l$是输入层，$L_{nl}$是输出层，中间为隐含层。

上图中神经网络的参数为：

$$(W,b) = (W^{(1)}, b^{(1)}, W^{(2)}, b^{(2)})$$

$W^{(l)}_{ij}$表示$l$层到$l+1$层的权值矩阵或向量，$i$代表$l+1$层，$j$代表$l$层的索引。
$W^{(1)} \in \Re^{3\times 3}$
$\ W^{(2)} \in \Re^{1\times 3}$
$a^{(l)}_i$ 表示第 $i$ 的激活函数的结果 $l$
对于输入层即$l=1$， $a^{(1)}_i = x_i$，$i表示一个样本的第i列$

递推公式为：

$$a_{i}^{l+1}=f(z_{i}^{l+1})=f(W_{i\cdot}^{l}x+b^l)$$
从输入层到输出层的计算详细过程如下：
$$\begin{eqnarray} a_{1}^{2} &=& f(W_{11}^{1}x_1 +W_{12}^{1}x_2+W_{13}^{1}x_3 + b_{1}^{1}) \\a_{2}^{2 } &=& f(W_{21}^{1}x_1 + W_{22}^{1}x_2 + W_{23}^{1}x_3+b_{2}^{1}) \\a_{3}^{2} &=& f(W_{31}^{1}x_1 + W_{32}^{1}x_2 + W_{33}^{1}x_3+b_{3}^{1}) \\h_{W,b}(x)=a_{1}^{3}&=&f(W_{11}^{2}x_1 + W_{12}^{2}x_2 + W_{13}^{2}x_3+b_{1}^{2}) \end{eqnarray}$$

输出层为多层时，神经网络的示意图如下：

---

3、backpropagation algorithm

每个样本对应的损失为：

$$\begin{align} J(W,b; x,y) = \frac{1}{2} \left\| h_{W,b}(x) - y \right\|^2. \end{align}$$
整体的损失函数加上对权重系数的正则化
$$\begin{eqnarray} J(W,b) &=& \left[ \frac{1}{m}\sum_{i=1}^{m}J(W,b,x^{i},y^{i})\right] + \frac{\lambda}{2}\sum_{l=1}^{n_l -1}\sum_{i=1}^{s_l}\sum_{j=1}^{s_l+1}(W_{ji}^{(l)})^2 \\ &=& \left[ \frac{1}{m}\sum_{i=1}^{m}( \frac{1}{2} \left\| h_{W,b}(x) - y \right\|^2)\right] + \frac{\lambda}{2}\sum_{l=1}^{n_l -1}\sum_{i=1}^{s_l}\sum_{j=1}^{s_l+1}(W_{ji}^{(l)})^2 \end{eqnarray}$$

反向传播的算法过程

1、正向传播，计算每层的激励函数值

$$a_{i}^{l+1}=f(z_{i}^{l+1})=f(W_{i\cdot}^{l}x+b^l)$$
2、利用y值和激励函数值计算输出层的’error term’
$$\begin{align} \delta^{(n_l)}_i = \frac{\partial}{\partial z^{(n_l)}_i} \;\; \frac{1}{2} \left\|y - h_{W,b}(x)\right\|^2 = - (y_i - a^{(n_l)}_i) \cdot f'(z^{(n_l)}_i) \end{align}$$
3、利用初值W，计算隐含层的‘error term’
$$\delta^{(l)}_i = \left( \sum_{j=1}^{s_{l+1}} W^{(l)}_{ji} \delta^{(l+1)}_j \right) f'(z^{(l)}_i)$$
4、计算每个样本的偏导。
$$\begin{align} \nabla_{W^{(l)}} J(W,b;x,y) &= \delta^{(l+1)} (a^{(l)})^T, \\ \nabla_{b^{(l)}} J(W,b;x,y) &= \delta^{(l+1)}. \end{align}$$
5、计算整体的偏导
$$\begin{align} \frac{\partial}{\partial W_{ij}^{(l)}} J(W,b) &= \left[ \frac{1}{m} \sum_{i=1}^m \frac{\partial}{\partial W_{ij}^{(l)}} J(W,b; x^{(i)}, y^{(i)}) \right] + \lambda W_{ij}^{(l)} \\ \frac{\partial}{\partial b_{i}^{(l)}} J(W,b) &= \frac{1}{m}\sum_{i=1}^m \frac{\partial}{\partial b_{i}^{(l)}} J(W,b; x^{(i)}, y^{(i)}) \end{align}$$
6、梯度下降更新 $W,b$
$$\begin{align} W_{ij}^{(l)} &= W_{ij}^{(l)} - \alpha \frac{\partial}{\partial W_{ij}^{(l)}} J(W,b) \\ b_{i}^{(l)} &= b_{i}^{(l)} - \alpha \frac{\partial}{\partial b_{i}^{(l)}} J(W,b) \end{align}$$

梯度下降更新$W,b$

$$\begin{align} \frac{\partial}{\partial W_{ij}^{(l)}} J(W,b) &= \left[ \frac{1}{m} \sum_{i=1}^m \frac{\partial}{\partial W_{ij}^{(l)}} J(W,b; x^{(i)}, y^{(i)}) \right] + \lambda W_{ij}^{(l)} \\ \frac{\partial}{\partial b_{i}^{(l)}} J(W,b) &= \frac{1}{m}\sum_{i=1}^m \frac{\partial}{\partial b_{i}^{(l)}} J(W,b; x^{(i)}, y^{(i)}) \end{align}$$
$$\begin{eqnarray} \frac{\partial J(W,b;x,y)}{\partial W_{ji}^{l}}&=&\frac{ \partial J(W,b;x,y) }{\partial z_{j}^{(l+1)}}\cdot \frac{\partial z_{j}^{(l+1)}}{\partial W_{ji}^{l}} \\ &=& \frac{ \partial J(W,b;x,y) }{\partial z_{j}^{(l+1)}}\cdot \frac{\partial \sum_{i=1}^{n}W_{ji}^{(l)} a_{i}^{l} + b_{i}^{l}}{\partial W_{ji}^{l}} \\ &=& \frac{ \partial J(W,b;x,y) }{\partial z_{j}^{(l+1)}}\cdot a_{i}^{l} \end{eqnarray}$$
$$\begin{eqnarray} \delta_{j}^{(n_l)}&= & \frac{ \partial J(W,b;x,y) }{\partial z_{j}^{(n_l)}} \\&=&\frac{\partial \frac{1}{2}||y-h_{W,b}||^2}{\partial z_{j}^{(n_l)}} \\ &=& \frac{\partial \frac{1}{2}||y-a_{j}^{(n_l)}||^2}{\partial z_{j}^{(n_l)}} \\ &=& -(y-a_{j}^{(n_l)})\frac{\partial a_{j}^{(n_l)}}{\partial z_{j}^{(n_l)}} \\ &=& -(y-a_{j}^{(n_l)})f^{'}(z_{i}^{n_l}) \end{eqnarray}$$
$$\begin{eqnarray} \frac{\partial J(W,b;x,y)}{\partial W_{ji}^{l}}&=& -(y-a_{j}^{(n_l)})f^{'}(z_{i}^{n_l})\cdot a_{i}^{l} \\ &=& \delta_{j}^{(n_l)}\cdot a_{i}^{l} \end{eqnarray}$$
对于隐含层，其误差项通过其后面一层传播而来，对于第L层，相当于L+1 层有 $s_{l+1}项误差求和而来$
$$\begin{eqnarray} \delta_{i}^{l}&=& \frac{ \partial J(W,b;x,y)}{\partial z_{i}^{l}} \\ &=& \sum_{j=1}^{S_{l+1}}\frac{\partial J(W,b;x,y)}{\partial z_{j}^{l+1}}\frac{\partial z_{j}^{l+1}}{\partial z_{j}^{l}} \\ &=& \sum_{j=1}^{S_{l+1}}\frac{\partial J(W,b;x,y)}{\partial z_{j}^{l+1}}\frac{\partial (\sum_{i=1}^{n}W_{ji}^{l}a_{i}^{l}+b_{i}^{l})}{\partial z_{j}^{l}} \\&=& \sum_{j=1}^{S_{l+1}}\frac{\partial J(W,b;x,y)}{\partial z_{j}^{l+1}}\frac{\partial (\sum_{i=1}^{n}W_{ji}^{l}f(z_{i}^{l})+b_{i}^{l})}{\partial z_{j}^{l}} \\ &=& \sum_{j=1}^{S_{l+1}}\frac{\partial J(W,b;x,y)}{\partial z_{j}^{l+1}}W_{ji}^{l}f^{'}(z_{i}^{l}) \\ &=& \sum_{j=1}^{S_{l+1}} \delta_{j}^{(l+1)}W_{ji}^{l}f^{'}(z_{i}^{l}) \end{eqnarray}$$

---

4、softmax函数及其求导

输出层激励函数为softmax：

$$a_j^L= \frac{e^{z_j^L}}{\sum_ke^{z_k^L}}$$
交叉熵代价函数为：
$$C= -\sum_{k}y_k\ log\ a_k$$
$关于W和b的偏导为：$
$$\frac{\partial C}{\partial w_{jk}}=a_k^{L-1}(a_j^L-y_j)$$
$$\frac{\partial C}{\partial b_{j}}=a_j^L-y_j$$

• 公式推导过程
  softmax函数公式：
  $$a_j^L= \frac{e^{z_j^L}}{\sum_ke^{z_k^L}}$$
  其中
  $$\begin{eqnarray} if \ j &=& i \\ \frac{\partial a_j}{\partial z_j}&=&\frac{\partial}{\partial z_i}(\frac{e^{z_j}}{\sum_k e^{z_k}}) \\ &=&\frac{(e^{z_j})^{'}\sum_k e^{z_k}-e^{z_j} \ e^{z_j}}{(\sum_k e^{z_k})^2} \\&=& \frac{e^{z_j}}{\sum_k e^{z_k}}- \frac{e^{z_j}}{\sum_k e^{z_k}} \cdot \frac{e^{z_j}}{\sum_k e^{z_k}} \\ &=& a_j(1-a_j) \end{eqnarray}$$
  $$\begin{eqnarray} if \ j &\neq i \\ \frac{\partial a_j}{\partial z_j}&=&\frac{\partial}{\partial z_i}(\frac{e^{z_j}}{\sum_k e^{z_k}}) \\ &=& \frac{0\sum_k e^{z_k}-e^{z_j} \ e^{z_i}}{(\sum_k e^{z_k})^2} \\&=& - \frac{e^{z_j}}{\sum_k e^{z_k}} \cdot \frac{e^{z_i}}{\sum_k e^{z_k}} \\ &=& -a_ja_i \end{eqnarray}$$
  交叉熵代价函数为：
  $$C= -\sum_{k}y_k\ log\ a_k$$
  $$\begin{eqnarray} \frac{\partial C}{\partial b_j} &=& \frac{\partial C}{\partial z_j}\cdot \frac{\partial z_j}{\partial b_j} \\ &=& \frac{\partial C}{\partial z_j}\cdot \frac{\partial (w_{jk}a_k + b_j)}{\partial b_j} \\ &=& \frac{\partial}{\partial z_j}\left(-\sum_{k}y_k\ log\ a_k\right) \\ &=& -\sum_{k}y_k \frac{1}{a_k} \cdot \frac{\partial a_k}{\partial z_j} \\ &=&-y_j\frac{1}{a_j}a_j(1-a_j) -\sum_{k \neq j}y_k \frac{1}{a_k} \left(-a_ja_k\right) \\ &=& -y_j + a_j\sum_{k}y_k \\ &=& a_j -y_j \end{eqnarray}$$
  $$\frac{\partial C}{\partial w_{jk}}=a_k^{L-1}(a_j^L-y_j)$$

参考文献
http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/

http://blog.csdn.net/xuanyuansen/article/details/41214115

http://blog.csdn.net/u014313009/article/details/51045303