BP算法实例及代码实现

1. 计算图及参数设置

1.1 计算图

2022.10.17日更正：
参考其他网站，图中$W_{12}$和$W_{21}$应当位置互换，下角标的后位$n$统一标识从$X_n$节点的输出。这样才能再后面矩阵$W_0$运算的时候保持统一。
感谢评论区的盆友指出问题！

• 偏置项简化省略
• h输入为全连接，输出经过sigmoid层，得到中间的logits记为$Z$，得到$\hat{Y}$值再sigmoid层激活一下。
• Loss function 选取MSE
• $\sigma ()$函数的导数${\sigma }^{\prime }(x)=\sigma (x)(1-\sigma (x))$

1.2 参数设置

$X=[0.35,0.9{]}^T$

$y_{true}=0.5$

$W0=\left[\begin{array}{ll}{w}_{11}& w_{12}\\ w_{21}& w_{22}\end{array}\right]=\left[\begin{array}{ll}0.1& 0.8\\ 0.4& 0.6\end{array}\right]$

$W_1=[w_{h1},w_{h2}]=[0.3,0.9]$

步长$\alpha =0.01$

2. 正向传播过程

1. $h=W_0\cdot X=[0.755,0.68{]}^T$
2. $Z=\sigma (h)=[0.680,0.664{]}^T$
3. $Z_y=W_h\cdot Z=[0.8014]$
4. $\hat{Y}=\sigma (Z_y)=0.6903$
5. $L=\frac{1}{2}(Y_{true}-\hat{Y}{)}^2=\frac{1}{2}(0.5-0.6903{)}^2=0.0181$

神经网络训练的核心思路：

根据Loss$\to$调节参数$W_0,W_1$等$\to$结合GD等方法沿着梯度下降方法

3. BP过程

第一步预想求得$\frac{\partial L}{\partial W_{h1}}$，根据路径回溯可以看到：

$\{\begin{array}{c}L=\frac{1}{2}{\left(y-\hat{y}\right)}^2\\ \hat{y}=\sigma (Z_y)\\ Z_y=W_{h1}{h}_1+W_{h2}{h}_2\end{array}$

所以根据链式法则可以得到：
$$\begin{array}{l}\frac{\partial L}{\partial w_{h1}}=\frac{\partial L}{\partial \hat{y}}\ast \frac{\partial \hat{y}}{\partial Z_y}\ast \frac{\partial Z_y}{\partial W_{h1}}\\ =\frac{1}{2}(Y_{true}-\hat{Y}{)}^2\ast \sigma (Z_y)\ast (1-\sigma (Z_y))\ast h_1\\ =(0.6903-0.5)\ast (0.69)\ast (1-0.69)\ast 0.68\\ =0.02768\end{array}$$
第二步骤继续回溯，要想求得$\frac{\partial L}{\partial W_{11}}$，根据路径回溯可以看到：

$\{\begin{array}{c}\cdots \\ Z_1=\sigma (h_1)\\ h_1=W_{11}{X}_1+W_{21}{X}_2\end{array}$

所以根据链式法则可以得到：
$$\begin{array}{l}\frac{\partial L}{\partial w_{11}}=\frac{\partial L}{\partial \hat{y}}\ast \frac{\partial \hat{y}}{\partial Z_y}\ast \frac{\partial Z_y}{\partial Z_1}\ast \frac{\partial Z_1}{\partial h_1}\ast \frac{\partial h_1}{\partial W_{11}}\\ =\frac{1}{2}(Y_{true}-\hat{Y}{)}^2\ast \sigma (Z_y)\ast (1-\sigma (Z_y))\ast W_{h1}\ast \sigma (Z_1)\ast (1-\sigma (Z_1))\ast X_1\\ =(0.6903-0.5)\ast (0.69)\ast (1-0.69)\ast 0.3\ast 0.68\ast (1-0.68)\ast 0.35\\ =0.00093\end{array}$$

同理可得

$\nabla L_{W0}=\left[\begin{array}{ll}\frac{\partial L}{\partial W_{11}}& \frac{\partial L}{\partial W_{12}}\\ \frac{\partial L}{\partial W_{21}}& \frac{\partial L}{\partial W_{22}}\end{array}\right]=\left[\begin{array}{ll}0.00093& 0.002861\\ 0.002392& 0.00736\end{array}\right]$

$\nabla L_{W1}=[\frac{\partial L}{\partial W_{h1}},\frac{\partial L}{\partial W_{h2}}]=[0.02768,0.02703{]}^T$

所以，结合梯度下降法$e.g.W_{11}^{\prime }=W_{11}-\alpha \cdot \frac{\partial L}{\partial W_{11}}=0.99061$得到更新后的权重矩阵为

$W_0^{\prime }=W_0-\alpha \nabla L_{W0}=\left[\begin{array}{ll}0.99061& 0.79997\\ 0.399976& 0.599926\end{array}\right]$

$W_1^{\prime }=W_1-\alpha \nabla L_{W1}=[0.299972,0.899973{]}^T$

4. 用Numpy手撸Code

import numpy as np
import matplotlib.pylab as plt

def sigmoid_derive(x,derive=False):
	## 用于计算sigmoid函数的值或其求导函数的值
	if derive == True:
	    return x*(1-x)
	else:
	    return 1/(1+np.exp(-x))

X = np.array([[0.35],[0.9]]) #输入层
y = np.array([[0.5]]) #groundtruth
epochs = 200

np.random.seed(1)

W0 = np.array([[0.1,0.8],[0.4,0.6]])
W1 = np.array([[0.3,0.9]])

print("original:\n","W0:\n",W0,"\n W1:\n",W1)
loss = []
for epoch in range(epochs):
	print("In the process of %sth epoch"%(epoch+1))
	l0=X
	l1 = sigmoid_derive(np.dot(W0,l0)) # 计算隐藏层的输出
	l2 = sigmoid_derive(np.dot(W1,l1)) # 计算输出节点的logits
	l2_error = y-l2 #计算MSE Loss
	Loss = 0.5*np.abs(l2_error)**0.5
	loss.append(Loss[0][0])
	print("The Current Loss:", 0.5*np.abs(l2_error)**0.5)
	l2_delta = l2_error*sigmoid_derive(l2,derive=True) # 用于BackProp
	l1_error = l2_delta*W1
	l1_delta = l1_error*sigmoid_derive(l1,derive=True)
	W1 += l2_delta*l1.T
	W0 += l0.T.dot(l1_delta)
	print("After  BackProp:\n","W0:\n",W0,"\n W1:\n",W1)
	print('=========================================')

plt.plot(loss)
plt.xlabel('Epochs') 
plt.ylabel('Loss') 
plt.title('Loss Decreaing')
plt.show()

输出结果：

In the process of 1th epoch
The Current Loss: [[0.21810748]]
After  BackProp:
 W0:
 [[0.09661944 0.78985831]
 [0.39661944 0.58985831]]
 W1:
 [[0.27232597 0.87299836]]
=========================================
In the process of 2th epoch
The Current Loss: [[0.21319219]]
After  BackProp:
 W0:
 [[0.09363393 0.78028763]
 [0.39363393 0.58028763]]
 W1:
 [[0.2455836  0.84691021]]
=========================================
In the process of 3th epoch
The Current Loss: [[0.20830283]]
After  BackProp:
 W0:
 [[0.09102066 0.7712756 ]
 [0.39102066 0.5712756 ]]
 W1:
 [[0.21978966 0.82175133]]

训练Loss降低示意图：