全连接神经网络基础——反向传播及梯度下降

上文(传送门)说到全连接神经网络的正向传播以及损失函数，通过损失函数值来评价网络的拟合效果，如何实现在训练过程中降低损失函数值呢？就涉及到本文的主要内容，反向传播以及梯度下降了。

反向传播

还是以这个网络结构为例，

通过正向传播以及损失函数，我们获得损失函数关于输入$\mathbf{x}$、权重$\mathbf{W}$和偏置$\mathbf{B}$的复合函数，即$$L=L(\hat{y},y)=L(f(\mathbf{x},\mathbf{W},\mathbf{B}),y)。$$反向传播过程中，我们需要计算出损失函数$L$对权重$\mathbf{W}$和偏置$\mathbf{B}$的偏导数，以便于进行梯度下降。

根据不同的损失函数设置，可以先计算得到损失函数对网络输出的偏导数$\frac{\partial L}{\partial \hat{y}}$。

输出层:$$\hat{y}=f_3({\mathbf{W}}^{(3)}{\mathbf{x}}^{(3)}+{\mathbf{B}}^{(3)})=\delta (w_{11}^{(3)}{x}_{31}+w_{12}^{(3)}{x}_{32}+w_{13}^{(3)}{x}_{33}+b_1^{(3)}),$$这里的激活函数以sigmoid函数为例，由于$$\frac{\partial \delta (x)}{\partial x}=-\frac{1}{(1+e^{-x}{)}^2}(-e^{-x})=\frac{1+e^{-x}}{(1+e^{-x}{)}^2}-\frac{1}{(1+e^{-x}{)}^2}=\delta (x)(1-\delta (x)),$$因此根据链式法则，可以计算$$\begin{array}{rl}\frac{\partial L}{\partial w_{1i}^{(3)}}& =\frac{\partial L}{\partial \hat{y}}\frac{\partial \hat{y}}{w_{1i}^{(3)}}\\ & =\frac{\partial L}{\partial \hat{y}}\hat{y}(1-\hat{y})x_{3i},\forall i=1,\dots ,3，\end{array}$$因此$$\begin{array}{rl}\frac{\partial L}{\partial {\mathbf{W}}^{(3)}}& =[\frac{\partial L}{\partial w_{11}^{(3)}},\frac{\partial L}{\partial w_{12}^{(3)}},\frac{\partial L}{\partial w_{13}^{(3)}}]\\ & =\frac{\partial L}{\partial \hat{y}}\hat{y}(1-\hat{y})\times [x_{31},x_{32},x_{33}]\\ & =\frac{\partial L}{\partial \hat{y}}\hat{y}(1-\hat{y}){{\mathbf{x}}^{(3)}}^T。\end{array}$$类似的有$$\begin{array}{rl}\frac{\partial L}{\partial x_{3i}}& =\frac{\partial L}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial x_{3i}}\\ & =\frac{\partial L}{\partial \hat{y}}\hat{y}(1-\hat{y})w_{1i}^{(3)},\forall i=1,\dots ,3,\end{array}$$故$$\begin{array}{rl}\frac{\partial L}{\partial {\mathbf{x}}^{(3)}}& =[\frac{\partial L}{\partial x_{31}},\frac{\partial L}{\partial x_{32}},\frac{\partial L}{\partial x_{33}}{]}^T\\ & =\frac{\partial L}{\partial \hat{y}}\hat{y}(1-\hat{y})\times [w_{11}^{(3)},w_{12}^{(3)},w_{13}^{(3)}{]}^T\\ & =\frac{\partial L}{\partial \hat{y}}\hat{y}(1-\hat{y}){{\mathbf{W}}^{(3)}}^T。\end{array}$$同理
$$\frac{\partial L}{\partial {\mathbf{B}}^{(3)}}=\frac{\partial L}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial {\mathbf{B}}^{(3)}}=\frac{\partial L}{\partial \hat{y}}\hat{y}(1-\hat{y})。$$

第二个隐藏层：$${\mathbf{x}}^{(3)}=f_2({\mathbf{W}}^{(2)}{\mathbf{x}}^{(2)}+{\mathbf{B}}^{(2)}),$$即$$\begin{gathered} x_{31}=\delta (w_{11}^{(2)}{x}_{21}+w_{12}^{(2)}{x}_{22}+w_{13}^{(2)}{x}_{23}+b_1^{(2)}), \\ {x}_{32}=\delta (w_{21}^{(2)}{x}_{21}+w_{22}^{(2)}{x}_{22}+w_{23}^{(2)}{x}_{23}+b_2^{(2)}), \\ {x}_{33}=\delta (w_{31}^{(2)}{x}_{21}+w_{32}^{(2)}{x}_{22}+w_{33}^{(2)}{x}_{23}+b_3^{(2)}). \end{gathered}$$可以看出$w_{11}^{(2)}$仅通过$x_{21}$对$x_{31}$有梯度贡献，因此$$\frac{\partial L}{\partial w_{11}^{(2)}}=\frac{\partial L}{\partial x_{31}}\frac{\partial x_{31}}{\partial w_{11}^{(2)}}=\frac{\partial L}{\partial x_{31}}{x}_{31}(1-x_{31})x_{21},$$故对于$\forall i=1,\dots ,3$和$j=1,\dots ,3$，有$$\frac{\partial L}{\partial w_{ij}^{(2)}}=\frac{\partial L}{\partial x_{3i}}\frac{\partial x_{3i}}{\partial w_{ij}^{(2)}}=\frac{\partial L}{\partial x_{3i}}{x}_{3i}(1-x_{3i})x_{2j},$$因此$$\begin{array}{rl}\frac{\partial L}{\partial {\mathbf{W}}^{(2)}}& =[\frac{\partial L}{\partial x_{3i}}{x}_{3i}(1-x_{3i})x_{2j}{]}_{3\times 3}\\ & =(\frac{\partial L}{\partial {\mathbf{x}}^{(3)}}\odot {\mathbf{x}}^{(3)}\odot (1-{\mathbf{x}}^{(3)})){{\mathbf{x}}^{(2)}}^T。\end{array}$$而对于$x_{21}$来说，分别通过$w_{11}^{(2)},w_{21}^{(2)},w_{31}^{(2)}$对$x_{31},x_{32},x_{33}$有梯度贡献，因此$$\begin{array}{rl}\frac{\partial L}{\partial x_{21}}& =\frac{\partial L}{\partial x_{31}}\frac{\partial x_{31}}{\partial x_{21}}+\frac{\partial L}{\partial x_{32}}\frac{\partial x_{32}}{\partial x_{21}}+\frac{\partial L}{\partial x_{33}}\frac{\partial x_{33}}{\partial x_{21}}\\ & =\frac{\partial L}{\partial x_{31}}{x}_{31}(1-x_{31})w_{11}^{(2)}+\frac{\partial L}{\partial x_{32}}{x}_{32}(1-x_{32})w_{21}^{(2)}+\frac{\partial L}{\partial x_{33}}{x}_{33}(1-x_{33})w_{31}^{(2)}\\ & =(\frac{\partial L}{\partial {\mathbf{x}}^{(3)}}\odot {\mathbf{x}}^{(3)}\odot (1-{\mathbf{x}}^{(3)}){)}^T[w_{11}^{(2)},w_{21}^{(2)},w_{31}^{(2)}{]}^T,\end{array}$$故对$\forall i=1,\dots ,3$，有$$\begin{array}{rl}\frac{\partial L}{\partial x_{2i}}& =\frac{\partial L}{\partial x_{31}}{x}_{31}(1-x_{31})w_{1i}^{(2)}+\frac{\partial L}{\partial x_{32}}{x}_{32}(1-x_{32})w_{2i}^{(2)}+\frac{\partial L}{\partial x_{33}}{x}_{33}(1-x_{33})w_{3i}^{(2)}\\ & =(\frac{\partial L}{\partial {\mathbf{x}}^{(3)}}\odot {\mathbf{x}}^{(3)}\odot (1-{\mathbf{x}}^{(3)}){)}^T[w_{1i}^{(2)},w_{2i}^{(2)},w_{3i}^{(2)}{]}^T,\end{array}$$因此$$\begin{array}{rl}\frac{\partial L}{\partial {\mathbf{x}}^{(2)}}& =[(\frac{\partial L}{\partial {\mathbf{x}}^{(3)}}\odot {\mathbf{x}}^{(3)}\odot (1-{\mathbf{x}}^{(3)}){)}^T[w_{1i}^{(2)},w_{2i}^{(2)},w_{3i}^{(2)}{]}^T{]}_{3\times 1}\\ & =((\frac{\partial L}{\partial {\mathbf{x}}^{(3)}}\odot {\mathbf{x}}^{(3)}\odot (1-{\mathbf{x}}^{(3)}){)}^T{\mathbf{W}}^{(2)}{)}^T\\ & ={{\mathbf{W}}^{(2)}}^T((\frac{\partial L}{\partial {\mathbf{x}}^{(3)}}\odot {\mathbf{x}}^{(3)}\odot (1-{\mathbf{x}}^{(3)}))。\end{array}$$类似的$$\frac{\partial L}{\partial {\mathbf{B}}^{(2)}}=\frac{\partial L}{\partial {\mathbf{x}}^{(3)}}\odot {\mathbf{x}}^{(3)}\odot (1-{\mathbf{x}}^{(3)})。$$

第一个隐藏层：$${\mathbf{x}}^{(2)}=f_1({\mathbf{W}}^{(1)}\mathbf{x}+{\mathbf{B}}^{(1)}),$$即$$\begin{gathered} x_{21}=\delta (w_{11}^{(1)}{x}_{11}+w_{12}^{(1)}{x}_{12}+b_1^{(1)}), \\ {x}_{22}=\delta (w_{21}^{(1)}{x}_{11}+w_{22}^{(1)}{x}_{12}+b_2^{(1)}), \\ {x}_{23}=\delta (w_{31}^{(1)}{x}_{11}+w_{32}^{(1)}{x}_{12}+b_3^{(1)}). \end{gathered}$$同第二个隐藏层推到类似，对于$\forall i=1,\dots ,3$和$j=1,2$，有$$\begin{array}{rl}\frac{\partial L}{\partial w_{ij}}& =\frac{\partial L}{\partial x_{2i}}\frac{\partial x_{2i}}{\partial w_{ij}}\\ & =\frac{\partial L}{\partial x_{2i}}{x}_{2i}(1-x_{2i})x_{1j},\end{array}$$故$$\begin{array}{rl}\frac{\partial L}{\partial {\mathbf{W}}^{(1)}}& =[\frac{\partial L}{\partial x_{2i}}{x}_{2i}(1-x_{2i})x_{1j}{]}_{3\times 2}\\ & =(\frac{\partial L}{\partial {\mathbf{x}}^{(2)}}\odot {\mathbf{x}}^{(2)}\odot (1-{\mathbf{x}}^{(2)})){\mathbf{x}}^T。\end{array}$$同样的，对于$\forall i=1,2$，有$$\begin{array}{rl}\frac{\partial L}{\partial x_{1i}}& =\frac{\partial L}{\partial x_{21}}\frac{\partial x_{21}}{\partial x_{1i}}+\frac{\partial L}{\partial x_{22}}\frac{\partial x_{22}}{\partial x_{1i}}+\frac{\partial L}{\partial x_{23}}\frac{\partial x_{23}}{\partial x_{1i}}\\ & =\frac{\partial L}{\partial x_{21}}{x}_{21}(1-x_{21})w_{1i}^{(1)}+\frac{\partial L}{\partial x_{22}}{x}_{22}(1-x_{22})w_{2i}^{(1)}+\frac{\partial L}{\partial x_{23}}{x}_{23}(1-x_{23})w_{3i}^{(1)}\\ & =(\frac{\partial L}{\partial {\mathbf{x}}^{(2)}}\odot {\mathbf{x}}^{(2)}\odot (1-{\mathbf{x}}^{(2)}){)}^T[w_{1i}^{(1)},w_{2i}^{(1)},w_{3i}^{(1)}{]}^T\end{array}$$故$$\begin{array}{rl}\frac{\partial L}{\partial \mathbf{x}}& =[(\frac{\partial L}{\partial {\mathbf{x}}^{(2)}}\odot {\mathbf{x}}^{(2)}\odot (1-{\mathbf{x}}^{(2)}){)}^T[w_{1i}^{(1)},w_{2i}^{(1)},w_{3i}^{(1)}{]}^T{]}_{2\times 1}\\ & =((\frac{\partial L}{\partial {\mathbf{x}}^{(2)}}\odot {\mathbf{x}}^{(2)}\odot (1-{\mathbf{x}}^{(2)}){)}^T{\mathbf{W}}^{(1)}{)}^T\\ & ={{\mathbf{W}}^{(1)}}^T(\frac{\partial L}{\partial {\mathbf{x}}^{(2)}}\odot {\mathbf{x}}^{(2)}\odot (1-{\mathbf{x}}^{(2)}))。\end{array}$$同样的$$\begin{array}{r}\frac{\partial L}{\partial {\mathbf{B}}^{(1)}}=\frac{\partial L}{\partial {\mathbf{x}}^{(2)}}\odot {\mathbf{x}}^{(2)}\odot (1-{\mathbf{x}}^{(2)})。\end{array}$$

梯度下降

在网络的训练过程中，主要有三种梯度下降方法，分别是批量梯度下降、随机梯度下降、小批量梯度下降。

批量梯度下降：每次迭代中使用所有训练数据进行梯度更新，以上文讲到的MSE损失函数为例，$L$对网络参数$w$的偏导可以描述为$$\frac{\partial L}{\partial w}=\frac{2}{n}\sum _{i=1}^n(y_i-\widehat{y_i})\frac{\partial \widehat{y_i}}{\partial w}，$$可以发现$n$个样本都对参数$w$的梯度产生了贡献，即将每个样本单独计算得到的梯度计算均值作为网络参数更新的梯度。

随机梯度下降：批量梯度下降有个明显的缺陷，即网络更新较慢，针对这个问题，随机梯度下降方法每次只使用一个样本进行更新网络参数，即$$\frac{\partial L}{\partial w}=2(y_i-\widehat{y_i})\frac{\partial \widehat{y_i}}{\partial w}。$$

小批量梯度下降：随机梯度下降虽然解决了批量梯度下降网络更新速度慢的问题，但是也产生了新的问题，如容易陷入局部最优以及不易实现并行化，对此小批量梯度下降使用部分样本进行网络的参数迭代更新。

在计算得到网络参数的梯度之后，沿着梯度的负方向，以一定步长（学习率）进行更网络参数，即$$w=w-\alpha \frac{\partial L}{\partial w},$$其中$\alpha$表示网络的学习率，是一个重要的超参数，如果学习率过大，容易导致网络震荡，过小则会导致网络收敛较慢。

除此之外，网络训练过程中还有一些其他常用的优化算法。
Momentum：通过记录历史梯度与当前梯度共同完成网络参数的迭代。
AdaGrad：网络参数的更新过程使用可变步长。
Adam：采用可变步长与记录历史梯度的方式更新网络参数。

下面通过一段代码实现本文最开始的网络结构，

import numpy as np

class Net:
    def __init__(self):
        self.W1 = np.random.random((3, 2))
        self.B1 = np.random.random((3, 1))
        self.W2 = np.random.random((3, 3))
        self.B2 = np.random.random((3, 1))
        self.W3 = np.random.random((1, 3))
        self.B3 = np.random.random((1, 1))
        self.rate = 0.1
        self.gradient = None
        self.x2 = None
        self.x3 = None
        self.y = None
        self.x = None
        self.label = None

    def train(self, x, label):
        self.label = label
        self.x = x
        for i in range(100):
            self.forward(x)
            loss = self.computeLoss()
            self.backward()
            self.graDesc(self.rate)
            if i % 20 == 0:
                print("迭代次数：", i, "预测值：", self.y, "损失函数值：", loss)
        print("迭代次数：", 99, "预测值：", self.y, "损失函数值：", loss)

    def sigmoid(self, x):
        return 1 / (1 + np.exp(-x))

    def forward(self, x):
        self.x2 = self.sigmoid(np.dot(self.W1, x) + self.B1)
        self.x3 = self.sigmoid(np.dot(self.W2, self.x2) + self.B2)
        self.y = self.sigmoid(np.dot(self.W3, self.x3) + self.B3)

    def computeLoss(self):
        return (self.y - self.label) ** 2

    def backward(self):
        dLdy = 2 * (self.y - self.label)
        dLdW3 = dLdy * self.y * (1 - self.y) * self.x3.T
        dLdB3 = dLdy * self.y * (1 - self.y)
        dLdx3 = dLdy * self.y * (1 - self.y) * self.W3.T
        dLdW2 = np.dot(dLdx3 * self.x3 * (1 - self.x3), self.x2.T)
        dLdB2 = dLdx3 * self.x3 * (1 - self.x3)
        dLdx2 = np.dot(self.W2.T, dLdx3 * self.x3 * (1 - self.x3))
        dLdW1 = np.dot(dLdx2 * self.x2 * (1 - self.x2), self.x.T)
        dLdB1 = dLdx2 * self.x2 * (1 - self.x2)
        self.gradient = [dLdW1, dLdB1, dLdW2, dLdB2, dLdW3, dLdB3]

    def graDesc(self, rate):
        self.W1 -= rate * self.gradient[0]
        self.B1 -= rate * self.gradient[1]
        self.W2 -= rate * self.gradient[2]
        self.B2 -= rate * self.gradient[3]
        self.W3 -= rate * self.gradient[4]
        self.B3 -= rate * self.gradient[5]

if __name__ == "__main__":
    n = Net()
    x = np.array([[5, 7]]).T
    label = np.array(0.6)
    n.train(x, label)

输出结果如下。

迭代次数： 0 预测值： [[0.81946645]] 损失函数值： [[0.04816552]]
迭代次数： 20 预测值： [[0.75549392]] 损失函数值： [[0.02417836]]
迭代次数： 40 预测值： [[0.69519441]] 损失函数值： [[0.00906198]]
迭代次数： 60 预测值： [[0.65235451]] 损失函数值： [[0.00274099]]
迭代次数： 80 预测值： [[0.62715023]] 损失函数值： [[0.00073713]]
迭代次数： 99 预测值： [[0.61416866]] 损失函数值： [[0.00020075]]

注：博主才疏学浅，若有纰漏，还请各位不吝赐教！