神经网络的前向传播和反向传播推导

神经网络的前向传播和反向传播推导

$x_1$和$x_2$表示输入
$w_{ij}$表示权重
$b_{ij}$表示偏置
${\sigma }_i$表示激活函数，这里使用sigmoid激活函数
$out$表示输出
$y$表示真实值
$\eta$表示学习率

前向传播
$h_1=w_{11}{x}_1+w_{13}{x}_2+b_{11}$，${\alpha }_1=\sigma (h1)=\frac{1}{1+e^{-h1}}$

$h_2=w_{12}{x}_1+w_{14}{x}_2+b_{12}$，${\alpha }_2=\sigma (h2)=\frac{1}{1+e^{-h2}}$

$z=w_{21}{\alpha }_1+w_{22}{\alpha }_2+b_{21}$，$out=\sigma (z)=\frac{1}{1+e^{-z}}$

损失函数

$E=\frac{1}{2}(out-y{)}^2$

反向传播
求导
$△w_{21}=\frac{\partial E}{\partial w_{21}}=\frac{\partial E}{\partial out}\frac{\partial out}{\partial z}\frac{\partial z}{\partial w_{21}}=(out-y)\sigma (z)(1-\sigma (z)){\alpha }_1$

$△w_{22}=\frac{\partial E}{\partial w_{22}}=\frac{\partial E}{\partial out}\frac{\partial out}{\partial z}\frac{\partial z}{\partial w_{22}}=(out-y)\sigma (z)(1-\sigma (z)){\alpha }_2$

$△b_{21}=\frac{\partial E}{\partial b_{21}}=\frac{\partial E}{\partial out}\frac{\partial out}{\partial z}\frac{\partial z}{\partial b_{21}}=(out-y)\sigma (z)(1-\sigma (z))$

更新$w_{21}、w_{22}、b_{21}$

$w_{21}=w_{21}-\eta △w_{21}$

$w_{22}=w_{22}-\eta △w_{22}$

$b_{21}=b_{21}-\eta △b_{21}$

求导

$△w_{12}=\frac{\partial E}{\partial out}\frac{\partial out}{\partial z}\frac{\partial z}{\partial {\alpha }_2}\frac{\partial {\alpha }_2}{\partial h2}\frac{\partial {\alpha }_2}{\partial h_2}\frac{\partial h_2}{\partial w_{12}}=(out-y)\sigma (z)(1-\sigma (z))w_{22}\sigma (h_2)(1-\sigma (h_2))x_1$

$△w_{14}=\frac{\partial E}{\partial out}\frac{\partial out}{\partial z}\frac{\partial z}{\partial {\alpha }_2}\frac{\partial {\alpha }_2}{\partial h2}\frac{\partial {\alpha }_2}{\partial h_2}\frac{\partial h_2}{\partial w_{14}}=(out-y)\sigma (z)(1-\sigma (z))w_{22}\sigma (h_2)(1-\sigma (h_2))x_2$

$△b_{12}=\frac{\partial E}{\partial out}\frac{\partial out}{\partial z}\frac{\partial z}{\partial {\alpha }_2}\frac{\partial {\alpha }_2}{\partial h2}\frac{\partial {\alpha }_2}{\partial h_2}\frac{\partial h_2}{\partial b_{12}}=(out-y)\sigma (z)(1-\sigma (z))w_{22}\sigma (h_2)(1-\sigma (h_2))$

$△w_{11}=\frac{\partial E}{\partial out}\frac{\partial out}{\partial z}\frac{\partial z}{\partial {\alpha }_1}\frac{\partial {\alpha }_1}{\partial h1}\frac{\partial {\alpha }_1}{\partial h_1}\frac{\partial h_1}{\partial w_{11}}=(out-y)\sigma (z)(1-\sigma (z))w_{21}\sigma (h_1)(1-\sigma (h_1))x_1$

$△w_{13}=\frac{\partial E}{\partial out}\frac{\partial out}{\partial z}\frac{\partial z}{\partial {\alpha }_1}\frac{\partial {\alpha }_1}{\partial h1}\frac{\partial {\alpha }_1}{\partial h_1}\frac{\partial h_1}{\partial w_{13}}=(out-y)\sigma (z)(1-\sigma (z))w_{21}\sigma (h_1)(1-\sigma (h_1))x_2$

$△b_{11}=\frac{\partial E}{\partial out}\frac{\partial out}{\partial z}\frac{\partial z}{\partial {\alpha }_1}\frac{\partial {\alpha }_1}{\partial h1}\frac{\partial {\alpha }_1}{\partial h_1}\frac{\partial h_1}{\partial b_{11}}=(out-y)\sigma (z)(1-\sigma (z))w_{21}\sigma (h_1)(1-\sigma (h_1))$

更新$w_{12}、w_{14}、b_{12}$

$w_{12}=w_{12}-\eta △w_{12}$

$w_{14}=w_{14}-\eta △w_{14}$

$b_{12}=b_{12}-\eta △b_{12}$

更新$w_{11}、w_{13}、b_{11}$

$w_{11}=w_{11}-\eta △w_{11}$

$w_{13}=w_{13}-\eta △w_{13}$

$b_{11}=b_{11}-\eta △b_{11}$

import matplotlib.pyplot as plt
import numpy as np

# 定义参数
# N：样本数量
# D_in：数据维度、输入维度
# H：隐藏层神经元个数
# D_out：输出维度
N, D_in, H, D_out = 64, 1000, 100, 10

# 生成数据
x = np.random.randn(D_in, N)
y = np.random.randn(D_out, N)

# 初始化参数
w1 = np.random.randn(D_in, H)
b1 = np.zeros((H, N))
w2 = np.random.randn(H, D_out)
b2 = np.zeros((D_out, N))

# 学习率
learning_rate = 1e-6

loss_list = []

# 最大跌打次数
iter = 500

for i in range(iter):
    # 前向传播
    h = np.matmul(w1.T, x)+b1 # (100, 64)
    a = np.maximum(h, 0) # (100, 64) relu激活函数
    y_pred = np.matmul(w2.T, a)+b2 # (10, 64)
    
    # 损失函数
    loss = np.square(y_pred-y).sum()
    
    loss_list.append(loss)
    
    # 反向传播
    grad_y_pred = 2*(y_pred-y) # (10, 64)
    grad_w2 = np.matmul(a, grad_y_pred.T) # (100, 10)
    grad_b2 = grad_y_pred # (10, 64)
    grad_a = np.matmul(w2, grad_y_pred) # (100, 64)
    grad_a[a<0] = 0
    grad_w1 = np.matmul(x, grad_a.T) # (1000, 100)
    grad_b1 = grad_a # (100, 64)
    
    # 更新参数
    w1 -= learning_rate*grad_w1
    b1 -= learning_rate*grad_b1
    w2 -= learning_rate*grad_w2
    b2 -= learning_rate*grad_b2
    
plt.plot(range(iter), loss_list)
plt.ylabel('loss')
plt.xlabel('iter')
plt.show()