Python 手写 BP神经网络

误差反向传播算法

输出层

对训练例 $\left({\mathbf{x}}_k,{\mathbf{y}}_k\right)$, 假定神经网络的输出为 ${\hat{\mathbf{y}}}_k=\left({\hat{y}}_1^k,{\hat{y}}_2^k,\dots ,{\hat{y}}_l^k\right)$, 即
$${\hat{y}}_j^k=f\left({\beta }_j-{\theta }_j\right),$$
则网络在 $\left({\mathbf{x}}_k,{\mathbf{y}}_k\right)$ 上的均方误差为
$$E_k=\frac{1}{2}\sum _{j=1}^l{\left({\hat{y}}_j^k-y_j^k\right)}^2.$$
BP 算法基于梯度下降(gradient descent)策略, 以目标的负梯度方向对参数 进行调整. 对误差 $E_k$, 给定学习率 $\eta$, 有
$$\Delta w_{hj}=-\eta \frac{\partial E_k}{\partial w_{hj}}.$$

注意到 $w_{hj}$ 先影响到第 $j$ 个输出层神经元的输入值 ${\beta }_j$, 再影响到其输出值 ${\hat{y}}_j^k$, 然后影响到 $E_k$,那么根据链式法则有,
$$\frac{\partial E_k}{\partial w_{hj}}=\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial w_{hj}}.$$
因为有${\beta }_j=\sum _{h=1}^q{w}_{hj}{b}_h$
我们将${\beta }_j$抽象为斜率为$b_h$的一条直线,那么自然有
$$\frac{\partial {\beta }_j}{\partial w_{hj}}=b_h.$$

$$g_j=-\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}$$
$$=-\left({\hat{y}}_j^k-y_j^k\right)f^{\prime }\left({\beta }_j-{\theta }_j\right)$$
$$={\hat{y}}_j^k\left(1-{\hat{y}}_j^k\right)\left(y_j^k-{\hat{y}}_j^k\right).$$

结合上式得 $\Delta w_{hj}$
$$\Delta w_{hj}=\eta g_j{b}_h.$$
类似可得

$$\Delta {\theta }_j=-\eta g_j,$$

$$\Delta {\gamma }_h=-\eta e_h,$$

隐藏层

同理得出
$$\begin{array}{c}\begin{array}{rl}{e}_h=& -\frac{\partial E_k}{\partial b_h}\cdot \frac{\partial b_h}{\partial {\alpha }_h}\\ =& -\sum _{j=1}^l\frac{\partial E_k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}{f}^{\prime }\left({\alpha }_h-{\gamma }_h\right)\\ =& f^{\prime }\left({\alpha }_h-{\gamma }_h\right)\sum _{j=1}^l{w}_{hj}{g}_j\end{array}\end{array}$$
$$\Delta v_{ih}=\eta e_h{x}_i,$$
$$\Delta {\gamma }_h=-\eta e_h,$$

代码实现

from sklearn.model_selection import train_test_split
import numpy as np
import pandas as pd
from sklearn import datasets


iris = datasets.load_iris()
data = iris.data
target = iris.target

class NeuralNetwork:
    def __init__(self, in_size, o_size, h_size):
        # 初始化层的数量
        self.in_size = in_size
        self.o_size = o_size
        self.h_size = h_size
        
        self.W1 = np.random.randn(in_size, h_size) # n x b的矩阵
        self.W2 = np.random.randn(h_size, o_size) # b x k的矩阵
        
    def sigmod(self, x):
        return 1 / (1 + np.exp(-x))
    
    # 映射函数,将连续值变成离散值
    def ref(self, x):
        if x <= (1 / 3):
            return 0
        elif x <= (2 / 3):
            return 1
        else:
            return 2
        
    # 设输入X为 m x n的矩阵
    def forward(self, X):
        vec_rule = np.vectorize(self.ref)
        self.z2 = np.dot(X, self.W1) # m x b
        self.act2 = self.sigmod(self.z2)
        self.z3 = np.dot(self.act2, self.W2)# m x k
        self.y_hat = self.sigmod(self.z3)
        self.y_hat = vec_rule(self.y_hat)
        
        return self.y_hat
    # 设y为 m x k 的矩阵
    def backward(self, X, y, y_hat, leraning_rate):
        # 算出输出层的梯度顶
        Grd_1 = (y - y_hat) *  self.sigmod(self.z3) * (1 - self.sigmod(self.z3)) # m x k
        # 输出层的Δ值
        Delta_W2 = np.dot(self.act2.T, Grd_1) # b x k
        # 隐藏层的梯度顶
        Grd_2 = np.dot(Grd_1, self.W2.T) * self.sigmod(self.z2) * (1 - self.sigmod(self.z2)) # m x b
        # 隐藏层的Δ值
        Delta_W1 = np.dot(X.T, Grd_2) # n x b
        
        # 更新权值
        self.W1 += leraning_rate * Delta_W1
        self.W2 += leraning_rate * Delta_W2
        
    def tarin(self, X, y, learning_rate, num_epochs):
        # 检查形状
        if(X.shape[0] != y.shape[0]):
            return -1;
        for i in range(1, num_epochs + 1):
            y_hat = self.forward(X)
            self.backward(X, y, self.y_hat, learning_rate)
        # 输出均方误差
            loss = np.mean((y - y_hat) ** 2)
            print(f"loss = {loss}, epochs/num_epochs:{i}/{num_epochs}")
    def predict(self, X):
        y_pred = self.forward(X)
        return y_pred
        

---

注: 部分公式来自周志华的西瓜书