深度学习——手写BP神经网络

---

一、前置数学公式

神经网络是由一个一个神经元组成，其基本结构如下：

无数以链式结构组成在一起的神经元构成了神经网络，设置合理的$w$和$b$可以拟合任何数学模型，训练神经网络的本质就是利用梯度下降求解最佳$w$和$b$。由于神经网络颇为复杂，无法直接求解某个参数的梯度，所以主流的方法是采用反向传播($BP$)的方法一步步前推，具体详见深度学习笔记(1)——神经网络详解及改进。

1、$cross$ $entropy$交叉熵

$$L(p,q)=-\sum _{i=1}^{n}p(x_i)\ast \log q(x_i)$$
交叉熵一般用作于分类问题的损失函数，其结果越小，$p(x)$和$q(x)$的分布越接近。

2、$sigmoid$

$$a=\sigma (x)=\frac{1}{1+e^{-x}}$$
$sigmoid$作为最简单的激活函数，在$x$趋近于负无穷时，$y$趋近于0，$x$趋近于正无穷时，$y$趋近于1。

• 对$sigmoid$求导
  $$a^{\prime }=\sigma (x)(1-\sigma (x))$$
• 损失函数$l$对$z$求导：$\partial l/\partial z$
  $$\frac{\partial l}{\partial z_i}={\sigma }^{\prime }(z)\sum w_i\frac{\partial l}{\partial z_{i+1}}$$

def sigmoid(z, driv=False):    # driv 微分
    if(driv == True):   # 微分（输入z，返回dz）
        return sigmoid(z)*(1-sigmoid(z))
    return 1/(1+np.exp(-z))

3、$softmax$

$$a=\frac{e^{z_i}}{{\sum }_j^n{e}^{z_j}}$$
$softmax$是指数除以指数之和，一般作为分类任务的输出层，输出结果可以认为是几个类别选择的概率。
例如三分类问题，$z_1=3,z_2=1,z_3=-3$，经过$softmax$后$a_1=0.88,a_2=0.12,a_3=0$。所以通俗得讲$softmax$的作用就是对输出的最大值进行强化，并将其归一化至$[0,1]$之间。

• 对$softmax$求导
  如果$i=j$：
  $$\frac{\partial a_i}{\partial z_i}=a_i(1-a_i)$$
  如果$i\ne j$：
  $$\frac{\partial a_j}{\partial z_i}=-a_i{a}_j$$

• 损失函数$l$对$z$求导：$\partial l/\partial z$
  $$\frac{\partial l}{\partial z_i}=\frac{\partial l}{\partial a_j}\frac{\partial a_j}{\partial z_i}=a_i-y_i$$

def softmax(self, z=None, a=None, y=None, driv=False):  # softmax层
	if(driv==True): # 损失函数对z的微分
	    return a-y
	size = z.shape[0]
	log_out = np.exp(z)    # 对输出结果取ln
	log_sum = np.sum(log_out,axis=1).reshape(size,1)   # sum(e^output)
    return np.exp(z)/log_sum

二、代码编写

1、数据形式

• 训练集$x$:
  $x$是一个$size\ast dimension$的二维数组，$size$代表训练集$x$的数量，$dimension$代表训练集的维度。
  $x={\left[\begin{array}{cccc}{x}_1^1& x_2^1& ...& x_d^1\\ x_1^2& x_2^2& ...& x_d^2\\ ...& ...& & ...\\ x_1^s& x_2^s& ...& x_d^s\end{array}\right]}_{size\ast dimension}$ ，其中$x_j^i$代表第$i$组数据的第$j$个输入值。

• 神经元值$a$:
  $a$是一个$layers\ast size\ast units$的三维数组，$layers$代表神经网络层数，$size$代表训练集数量，$units$代表该层网络有几个神经元。
  $a[i]={\left[\begin{array}{cccc}{a}_1^1& a_2^1& ...& a_u^1\\ a_1^2& a_2^2& ...& a_u^2\\ ...& ...& & ...\\ a_1^s& a_2^s& ...& a_u^s\end{array}\right]}_{size\ast units}$，其中$a[i][j][k]$代表第$i$层网络，第$j$组数据，第$k$个神经元的值。例：$a[1]=x$，第1层网络就是输入层。

• 未经过激活函数的神经元$z$
  数据形式和$a$相同。

• 损失函数$l$对$z$的微分$dz$
  数据形式和$a$相同。
  $dz[i]={\left[\begin{array}{cccc}d{z}_1^1& d{z}_2^1& ...& d{z}_u^1\\ d{z}_1^2& d{z}_2^2& ...& d{z}_u^2\\ ...& ...& & ...\\ d{z}_1^s& d{z}_2^s& ...& d{z}_u^s\end{array}\right]}_{size\ast units}$

• 偏差$b$
  $b$是一个$layers\ast units$的二维数组，$b[i][j]$代表第$i$层网络第$j$个神经元上的偏差。

• 权重$w$
  $w$是一个$(layers-1)\ast units1\ast units2$的三维数组，$w[i]$表示第$i$层和第$i+1$层之间的权值，$units1$是第$i$层的神经元个数，$units2$是第$i+1$层的神经元个数。
  $w[i]={\left[\begin{array}{cccc}{w}_{11}& w_{12}& ...& w_{1u_2}\\ w_{21}& w_{22}& ...& w_{2u_2}\\ ...& ...& & ...\\ w_{u_{1}1}& w_{u_{1}2}& ...& w_{u_1{u}_2}\end{array}\right]}_{units1\ast units2}$，$w[i][j][k]$代表第$i$层第$j$个神经元和第$i+1$层第$k$个神经元之间的权重。

1、前向传播

比较简单,每一层的值等于前一层的值矩阵乘权重加上$bias$，用下面这个三层网络（输入层+隐藏层+输出层）作例子。

$$z=\left[\begin{array}{c}{x}_1^1\\ x_1^2\\ ...\\ x_1^s\end{array}\right]\ast w_1+\left[\begin{array}{c}{x}_2^1\\ x_2^2\\ ...\\ x_2^s\end{array}\right]\ast w_2+b$$
用矩阵表达：
$${\left[\begin{array}{c}{z}^1\\ z^2\\ ...\\ z^s\end{array}\right]}_{size\ast 1}={\left[\begin{array}{cc}{x}_1^1(a_1^1)& x_2^1(a_2^1)\\ x_1^2(a_1^2)& x_2^2(a_2^2)\\ ...& ...\\ x_1^s(a_1^s)& x_2^s(a_2^s)\end{array}\right]}_{size\ast 2}\left[\begin{array}{c}{w}_1\\ w_2\end{array}\right]+b$$
公式扩展到每层网络：
$$z_{i+1}=a_i\ast w_i+b_i，a_{i+1}=\sigma (z_{i+1})$$

• $i$代表网络第$i$层数
• $a_{i+1}$：第$i+1$层网络的神经元值
• $w_i$：第$i$和$i+1$层网络之间的权值

def ForwardPropagation(x_train, num_layers): # 神经网络的结果
	'''
	@param num_layers:网络层数
	'''
    z = {}      # 未经过激活函数的神经元 [第x个层][第y个数据, 第z个神经元]
    a = {}      # 经过激活函数的神经元 [第x个层][第y个数据, 第z个神经元]
    a[0] = x_train
    for i in range(self.num_layers-2):
        z[i+1] = a[i]@w[i]+b[i]
        a[i+1] = sigmoid(z[i+1])
    # softmax层
    z[num_layers-1] = a[num_layers-2]@w[num_layers-2]+b[num_layers-2]
    a[num_layers-1] = softmax(z[num_layers-1])
    return z,a

2、反向传播

还是上面那张图。

$$\frac{dl}{dz}=\frac{dl}{da}\frac{da}{dz}=(w_3\ast d{z}^{\prime }+w_4\ast d{z}^{\prime \prime })\ast \frac{da}{dz}$$
用矩阵表达：
$${\left[\begin{array}{c}d{z}^1\\ d{z}^2\\ ...\\ d{z}^s\end{array}\right]}_{size\ast 1}={\left[\begin{array}{cc}d{z}^{\prime 1}& d{z}^{\prime \prime 1}\\ d{z}^{\prime 2}& d{z}^{\prime \prime 2}\\ ...& ...\\ d{z}^{\prime s}& d{z}^{\prime \prime s}\end{array}\right]}_{size\ast 2}\left[\begin{array}{cc}{w}_3& w_4\end{array}\right].T{\left[\begin{array}{c}{\frac{da}{dz}}^1\\ {\frac{da}{dz}}^2\\ ...\\ {\frac{da}{dz}}^s\end{array}\right]}_{size\ast 1}$$
也就相当于
$$d{z}_2=d{z}_3\ast w_2.T\ast \frac{d{a}_2}{d{z}_2}$$
扩展到每一层：
$$d{z}_i=d{z}_{i+1}\ast w_i\ast \frac{d{a}_i}{d{z}_i}$$

def BackPropagation(z, a, y_train):   # bp计算Loss对z的偏微分
    dz = {}     # dl/dz [第x个层][第y个数据, 第z个神经元]
    # softmax层
    dz[num_layers-1] = softmax(a[num_layers-1],y_train,driv=True)/y_train.shape[0]      
    # 隐藏层
    for i in range(num_layers-2,0,-1):
        dz[i] = dz[i+1]@w[i].T*sigmoid(z[i],driv=True)  
    return dz  

3、计算$dw$和$db$

需要每一组数据集$x$分开来算，最后结果求平均。依旧如图：
$$d{w}_3=\frac{dl}{d{w}_3}=mean({\left[\begin{array}{c}d{w}_3^1\\ d{w}_3^2\\ ...\\ d{w}_3^s\end{array}\right]}_{size\ast 1})=a\ast {\left[\begin{array}{c}d{z}^{\prime 1}\\ d{z}^{\prime 2}\\ ...\\ d{z}^{\prime s}\end{array}\right]}_{size\ast 1}$$
用矩阵表达：
$a[i]={\left[\begin{array}{cccc}{a}_1^1& a_2^1& ...& a_{u1}^1\\ a_1^2& a_2^2& ...& a_{u1}^2\\ ...& ...& & ...\\ a_1^s& a_2^s& ...& a_{u1}^s\end{array}\right]}_{size\ast units1},a[i][j]=\left[\begin{array}{cccc}{a}_1^j& a_2^j& ...& a_{u1}^j\end{array}\right]$

$dz[i]={\left[\begin{array}{cccc}d{z}_1^1& d{z}_2^1& ...& d{z}_{u2}^1\\ d{z}_1^2& d{z}_2^2& ...& d{z}_{u2}^2\\ ...& ...& & ...\\ d{z}_1^s& d{z}_2^s& ...& d{z}_{u2}^s\end{array}\right]}_{size\ast units2},dz[i][j]=\left[\begin{array}{cccc}d{z}_1^j& d{z}_2^j& ...& d{z}_{u2}^j\end{array}\right]$

用其中一组$x$得到的前一层的$a$和下一层的$dz$的矩阵乘：$a[i][j].T\ast dz[i+1][j]=\left[\begin{array}{c}{a}_1^j\\ a_2^j\\ ...\\ a_{u1}^j\end{array}\right]\left[\begin{array}{cccc}d{z}_1^j& d{z}_2^j& ...& d{z}_{u2}^j\end{array}\right]$得到一个$units1\ast units2$的矩阵（对应$w[i]$）。重复$s$次取平均值。

def cal_driv(dz,a):   # 计算dw和db
	dw =0
	db = np.mean(dz,axis=0)
	for i in range(dz.shape[0]):
	    dw = dw+a[[i]].T@dz[[i]]
	dw = dw/a.shape[0]
	return dw,db

4、梯度下降

没什么可说的：
$$w=w-dw\ast learningrate，b=b-db\ast learningrate$$

def gradient_decent(x_train, y_train):
    size = x_train.shape[0]
    z,a = ForwardPropagation(x_train)     # 进行一次前馈运算
    dz = BackPropagation(z,a,y_train)
    # 对 w,b进行梯度下降
    for i in range(self.num_layers-1):
        dw,db = cal_driv(dz[i+1],a[i])
        self.w[i] = self.w[i] - dw*learning_rate
        self.b[i] = self.b[i] - db*learning_rate

5、代码总览

import numpy as np

class BP_network():
    def __init__(self,input_dim, hidden_layer, output_dim):
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.w = {}                     # 利用字典存储 w    
        self.b = {}                     # 利用字典存储 bias
        self.num_layers = len(hidden_layer)+2   # 网络层数

        hidden_layer.insert(0,input_dim)
        hidden_layer.append(output_dim)
        for i in range(len(hidden_layer)-1):
            self.w[i] = np.random.rand(hidden_layer[i],hidden_layer[i+1])
            self.b[i] = np.zeros((1,hidden_layer[i+1]))

    def sigmoid(self, z, driv=False):    # driv 微分
        if(driv == True):   # 微分（输入z，返回dz）
            return self.sigmoid(z)*(1-self.sigmoid(z))
        return 1/(1+np.exp(-z))
    
    def softmax(self, z=None, a=None, y=None, driv=False):  # softmax层
        if(driv==True): # loss函数对z微分
            return a-y
        size = z.shape[0]
        log_out = np.exp(z)    # 对输出结果取ln
        log_sum = np.sum(log_out,axis=1).reshape(size,1)   # sum(e^output)
        return np.exp(z)/log_sum

    def fit(self, x_train, y_train, batch_size, learning_rate=0.01, epochs=20):
        def Loss(output, y_train): # 交叉熵
            return -np.sum(y_train*np.log(output))/output.shape[0]

        def ForwardPropagation(x_train): # 神经网络的结果
            z = {}      # 未经过激活函数的神经元 [第x个层][第y个数据, 第z个神经元]
            a = {}      # 经过激活函数的神经元 [第x个层][第y个数据, 第z个神经元]
            a[0] = x_train
            for i in range(self.num_layers-2):
                z[i+1] = a[i]@self.w[i]+self.b[i]
                a[i+1] = self.sigmoid(z[i+1])
            # softmax层
            z[self.num_layers-1] = a[self.num_layers-2]@self.w[self.num_layers-2]+self.b[self.num_layers-2]
            a[self.num_layers-1] = self.softmax(z[self.num_layers-1])
            return z,a

        def BackPropagation(z, a, y_train):   # bp计算Loss对z的偏微分
            dz = {}     # dl/dz [第x个层][第y个数据, 第z个神经元]
            # softmax层
            dz[self.num_layers-1] = self.softmax(a=a[self.num_layers-1],y=y_train,driv=True)/y_train.shape[0]      
            # 隐藏层
            for i in range(self.num_layers-2,0,-1):
                dz[i] = dz[i+1]@self.w[i].T*self.sigmoid(z[i],driv=True)  
            return dz  

        def cal_driv(dz,a):   # 计算dw和db
            dw = 0
            db = np.mean(dz,axis=0)
            for i in range(dz.shape[0]):
                dw = dw+a[[i]].T@dz[[i]]
            dw = dw/a.shape[0]
            return dw,db

        def gradient_decent(x_train, y_train):
            size = x_train.shape[0]
            z,a = ForwardPropagation(x_train)     # 进行一次前馈运算
            dz = BackPropagation(z,a,y_train)
            # 对 w,b进行梯度下降
            for i in range(self.num_layers-1):
                dw,db = cal_driv(dz[i+1],a[i])
                self.w[i] = self.w[i] - dw*learning_rate
                self.b[i] = self.b[i] - db*learning_rate

        size = x_train.shape[0]  # 数据集数量
        for i in range(epochs):
			for j in range(0,size,batch_size):
  			    gradient_decent(x_train[j:j+batch_size],y_train[j:j+batch_size])

            z,a = ForwardPropagation(x_train)
            loss = Loss(a[self.num_layers-1], y_train)   # 计算loss
            accuracy = self.accuracy(self.predict(x_train),y_train)
            print('第%d个epochs: Loss=%f，accuracy:%f' %(i+1,loss,accuracy))

    def predict(self, x_train):
        z = {}      # 未经过激活函数的神经元 [第x个层][第y个数据, 第z个神经元]
        a = {}      # 经过激活函数的神经元 [第x个层][第y个数据, 第z个神经元]
        a[0] = x_train
        for i in range(self.num_layers-2):
            z[i+1] = a[i]@self.w[i]+self.b[i]
            a[i+1] = self.sigmoid(z[i+1])
        # softmax层
        z[self.num_layers-1] = a[self.num_layers-2]@self.w[self.num_layers-2]+self.b[self.num_layers-2]
        a[self.num_layers-1] = self.softmax(z[self.num_layers-1])
        y = np.zeros(a[self.num_layers-1].shape)
        y[a[self.num_layers-1]-a[self.num_layers-1].max(axis=1).reshape(a[self.num_layers-1].shape[0],1)>=0]=1
        return y
        
    def accuracy(self,y,y_true):
        return np.sum(np.where(y==1)[1]==np.where(y_true==1)[1])/y.shape[0]

6、代码测试

准备数据集(简单的三分类数据集)：

x_train = np.random.randint(0,100,size=(100,2))
temp = (2*x_train[:,0]+x_train[:,1])
y_train = np.zeros((100,3))
y_train[temp>150,0] = 1
y_train[temp<50,1] = 1
y_train[((temp<=150) & (temp>=50)),2] = 1
x_train = (x_train-x_train.min(axis=0))/(x_train.max(axis=0)-x_train.min(axis=0))

训练网络

bp = BP_network(input_dim=2, hidden_layer=[5,5], output_dim=3)
bp.fit(x_train,y_train,batch_size=10,learning_rate=0.1,epochs=2000)

结果

第1个epochs: Loss=1.053458，accuracy:0.390000
第2个epochs: Loss=1.005622，accuracy:0.390000
第3个epochs: Loss=0.968565，accuracy:0.390000
第4个epochs: Loss=0.939970，accuracy:0.390000
第5个epochs: Loss=0.917880，accuracy:0.500000
第6个epochs: Loss=0.900729，accuracy:0.570000
第7个epochs: Loss=0.887299，accuracy:0.570000
第8个epochs: Loss=0.876668，accuracy:0.570000
第9个epochs: Loss=0.868147，accuracy:0.570000
第10个epochs: Loss=0.861226，accuracy:0.570000
...
第1997个epochs: Loss=0.561577，accuracy:0.830000
第1998个epochs: Loss=0.561012，accuracy:0.830000
第1999个epochs: Loss=0.560447，accuracy:0.830000
第2000个epochs: Loss=0.559881，accuracy:0.840000

代码正常运行