【CV基石】Forward Propagation 与 Backward Propagation（MSE简单版）

推导示例

前向传播

计算每一层的特征值X以及输出层和真值的偏差Loss。示例中采用Sigmoid作为激活函数，MSE作为Loss函数。

反向传播

不断的使用链式法则，从后向前求得Loss对每一层的权重W、偏置b的导数，对W、b进行更新，然后重复前传和反传操作，直到Loss收敛。

实现代码

Python实现代码如下：

# -*- coding: UTF-8 -*-
import numpy as np
from matplotlib import pyplot as plt
from pylab import mpl

# 设置中文可以显示
mpl.rcParams['font.sans-serif']=['SimHei']

# 激活函数
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# 损失函数
def mse_loss(out, label):
    N = out.shape[0]
    loss = np.sum(np.square(label - out)) / N
    return loss

# 前向传播 
def forward(X, W1, b1, W2, b2):
    Z1 = np.dot(W1, X) + b1
    A1 = sigmoid(Z1)
    Z2 = np.dot(W2, A1) + b2
    A2 = sigmoid(Z2)
    return Z1, A1, Z2, A2

# 反向传播
def backward(Y, A2, A1, X):
    dZ2 = -(Y - A2) * A2 * (1 - A2)
    dW2 = np.dot(dZ2, A1.T)
    db2 = 1 / 2 * np.sum(dZ2)
    dZ1 = np.dot(W2.T, dZ2) * A1 * (1-A1)
    dW1 = np.dot(dZ1, X.T)
    db1 = 1 / 3 * np.sum(dZ1)
    return dW2, db2, dW1, db1

# 梯度下降
def gradient_descent(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha):
    W1 = W1 - alpha * dW1
    b1 = b1 - alpha * db1
    W2 = W2 - alpha * dW2
    b2 = b2 - alpha * db2
    return W1, b1, W2, b2

# 神经网络训练
def NNTraining(X, W1, b1, W2, b2, Y, alpha, iterations):
    L = np.zeros((iterations, 1))
    for i in range(iterations):
        # 前向传播
        Z1, A1, Z2, A2 = forward(X, W1, b1, W2, b2)
        # 误差计算
        L[i] = mse_loss(Y, A2)
        # 反向传播
        dW2, db2, dW1, db1 = backward(Y, A2, A1, X)
        # 梯度下降
        W1, b1, W2, b2 = gradient_descent(W1, b1, W2, b2, dW1, db1, dW2, db2, alpha)
        print("iterations:", i, "loss:", L[i], "result:", A2[0], A2[1])
    return L
 
if __name__ == '__main__':
    # 输入
    X=np.array([[0.05],[0.10]])
    # 输出
    Y=np.array([[0.01],[0.99]])
    # 初始化权重和偏置
    W1=np.array([[0.15, 0.20], [0.25, 0.30], [0.05, 0.10]])
    b1=0.35  # 因为有广播机制，所以不用写成向量形式
    W2=np.array([[0.40, 0.45, 0.26], [0.50, 0.55, 0.13]])
    b2=0.6
    # 学习率
    alpha=0.5
    # 迭代次数
    iterations=10000

    L = NNTraining(X, W1, b1, W2, b2, Y, alpha, iterations)

    fig=plt.figure(1)
    plt.plot(L)
    plt.title(u'Loss曲线')
    plt.xlabel(u'iteration')
    plt.ylabel(u'loss')
    plt.show()

训练结果

iterations: 9990 loss: [2.14028324e-05] result: [0.01462654] [0.9853739]
iterations: 9991 loss: [2.13984463e-05] result: [0.01462606] [0.98537437]
iterations: 9992 loss: [2.13940613e-05] result: [0.01462559] [0.98537485]
iterations: 9993 loss: [2.13896775e-05] result: [0.01462512] [0.98537532]
iterations: 9994 loss: [2.13852949e-05] result: [0.01462464] [0.98537579]
iterations: 9995 loss: [2.13809135e-05] result: [0.01462417] [0.98537627]
iterations: 9996 loss: [2.13765332e-05] result: [0.01462369] [0.98537674]
iterations: 9997 loss: [2.13721542e-05] result: [0.01462322] [0.98537721]
iterations: 9998 loss: [2.13677763e-05] result: [0.01462275] [0.98537769]
iterations: 9999 loss: [2.13633995e-05] result: [0.01462227] [0.98537816]

参考链接

一文弄懂神经网络中的反向传播法——BackPropagation
python简单实现 反向传播算法
反向传播算法推导-卷积神经网络

---

Technical Exchange