感知器和神经网络训练（公式推导及C++实现）

感知器

感知器是一个二元线性分类器，其目标是找到一个超平面将训练样本进行分隔（分类），其表示为

一般使用的激活函数不是阶跃函数，常用的有sigmoid函数(其导数: $\sigma' = \sigma(1-\sigma)$)

在1个样本的训练集上代价函数（最小均方误差）通常为

$$C = \frac{1}{2} |a - y|^2$$
训练过程中步长(学习速率)为 $\alpha$的 $w$和$b$的新值为
$$\begin{align} w_i '&= w_i - \alpha \nabla_{w_i}C\\ b' &= b - \alpha \nabla_{b}C \end{align}$$
分别求其梯度，由于在神经元中，上一步的输出是下一步的输入，也就是下一步中的自变量，根据链式法则可以得到
$$\begin{align} \frac{\partial C}{\partial w_i} &= \frac{\partial C}{\partial z} \frac{\partial z}{\partial w_i}\\ \frac{\partial C}{\partial b} &= \frac{\partial C}{\partial z} \frac{\partial z}{\partial b} \end{align}$$
其中
$$\begin{align} \frac{\partial C}{\partial z} &= \frac{\partial C}{\partial a} \frac{\partial a}{\partial z} \\ &= \frac{1}{2} \frac{\partial}{\partial a}|a - y|^2\frac{\partial a}{\partial z}\\ & =(a - y)\sigma(1 - \sigma)\\ \\ \frac{\partial z}{\partial w_i} &= \frac{\partial}{\partial w_i}(\sum_{i=0}^{n-1}w_i x_i + b)\\ &= x_i\\ \frac{\partial z}{\partial b} &= \frac{\partial}{\partial w_i}(\sum_{i=0}^{n-1}w_i x_i + b)\\ &=1 \end{align}$$
最终可得
$$\begin{align} \frac{\partial C}{\partial w_i} &= \frac{\partial C}{\partial z} \frac{\partial z}{\partial w_i}\\ &= \frac{\partial C}{\partial a} \frac{\partial a}{\partial z} \frac{\partial z}{\partial w_i}\\ &=(a - y)\sigma(z)(1 - \sigma(z))x_i\\ \\ \frac{\partial C}{\partial b} &= \frac{\partial C}{\partial z} \frac{\partial z}{\partial b}\\ &= \frac{\partial C}{\partial a} \frac{\partial a}{\partial z} \frac{\partial z}{\partial b}\\ &=(a - y)\sigma(z)(1 - \sigma(z))\\ \end{align}$$

最终对有k个训练样本的感知机的迭代函数为

$$\begin{align} w_i '&= w_i - \alpha \frac{1}{k} \sum_{k=0}^{k-1} (a^k - y^k)\sigma(z)(1 - \sigma(z))x_i\\ b' &= b - \alpha \frac{1}{k} \sum_{k=0}^{k-1} (a^k - y^k)\sigma(z)(1 - \sigma(z)) \end{align}$$

多层神经网络

感知器可以看成一个最简单的二层神经网络，作为一个线性分类器它使用的局限性太大，而多层神经网络可以弥补它的缺点，并且训练原理与其类似

上图的三层神经网络多了一个隐含层，其中隐含层和输出层均为左下角的神经元，这个神经网络在训练过程时需要训练两层的神经网络，并且隐层的误差无法直接获取，而输出层的误差则和感知器相同：
同样在1个样本的训练集上代价函数（均方误差）通常为

$$C = \frac{1}{2}\sum_{i = 0}^{n-1}|a_i^j- y_i^j|^2$$
也就是输出层每个神经元的输出和对应的目标值误差的平方和。

训练输出层

因为只有输出层的误差可以获取，所以训练从输出层开始，对第三层的第一个来说

$$\begin{align} \frac{\partial C}{\partial w_{1i}^3} &= \frac{\partial C}{\partial a^3_1} \frac{\partial a_1^3}{\partial z_1^3}\frac{\partial z_1^3}{\partial w_{1i}^3}\\ &= \frac{\partial C}{\partial a^3_1} \frac{\partial a_1^3}{\partial z_1^3}x_{1i}^3\\ &= \frac{\partial C}{\partial a^3_1} \frac{\partial a_1^3}{\partial z_1^3}a_{1i}^2\\ \\ \frac{\partial C}{\partial b_1^3} &= \frac{\partial C}{\partial a^3_1} \frac{\partial a_1^3}{\partial z_1^3}\frac{\partial z_1^3}{\partial b_1^3}\\ &= \frac{\partial C}{\partial a^3_1} \frac{\partial a_1^3}{\partial z_1^3}\\ \end{align}$$

反向传播，训练隐含层

隐含层的训练也就是BP神经网络的重点，即误差的反向传播过程，因为隐含层的误差无法直接获取，所以才从输出层开始调整参数，然后误差向出入层开始传播，一层一层的进行训练

以训练隐含层的第一个神经元为例

$$\begin{align} \frac{\partial C}{\partial w_{1i}^2} &= \frac{\partial C}{\partial z^2_1} \frac{\partial z_1^2}{\partial w_{1i}2} \\ &= \frac{\partial C}{\partial a_1^2} \frac{\partial a_1^2}{\partial z^2_1} \frac{\partial z_1^2}{\partial w_{1i}2} \\ &= \frac{\partial C}{\partial z_1^3} \frac{\partial z_1^3}{\partial a_1^2} \frac{\partial a_1^2}{\partial z^2_1} \frac{\partial z_1^2}{\partial w_{1i}^2} + \frac{\partial C}{\partial z_2^3} \frac{\partial z_2^3}{\partial a_1^2} \frac{\partial a_1^2}{\partial z^2_1} \frac{\partial z_1^2}{\partial w_{1i}^2} \\ &= \frac{\partial C}{\partial a_1^3} \frac{\partial a_1^3}{\partial z_1^3} \frac{\partial z_1^3}{\partial a_1^2} \frac{\partial a_1^2}{\partial z^2_1} \frac{\partial z_1^2}{\partial w_{1i}^2} + \frac{\partial C}{\partial a_2^3} \frac{\partial a_2^3}{\partial z_2^3} \frac{\partial z_2^3}{\partial a_1^2} \frac{\partial a_1^2}{\partial z^2_1} \frac{\partial z_1^2}{\partial w_{1i}^2} \\ \end{align}$$

式子中红色部分即为输出层两个神经元计算过的，蓝色部分
$$z_1^3 = \sum w_1^3a_i^2 +b$$
$$\frac{\partial z_1^3}{\partial a_1^2} = w_1^3\\ \frac{\partial z_2^3}{\partial a_1^2} = w_2^3\\ $$
weight这些值都是确定的，最后黄色部分也很容易求得。
$$\begin{align} \frac{\partial C}{\partial w_{1i}^2} &= \frac{\partial C}{\partial a_1^3} \frac{\partial a_1^3}{\partial z_1^3} \frac{\partial z_1^3}{\partial a_1^2} \frac{\partial a_1^2}{\partial z^2_1} a_{1i}^1 + \frac{\partial C}{\partial a_2^3} \frac{\partial a_2^3}{\partial z_2^3} \frac{\partial z_2^3}{\partial a_1^2} \frac{\partial a_1^2}{\partial z^2_1}a_{1i}^1\\ \frac{\partial C}{\partial b_1^2} &= \frac{\partial C}{\partial a_1^3} \frac{\partial a_1^3}{\partial z_1^3} \frac{\partial z_1^3}{\partial a_1^2} \frac{\partial a_1^2}{\partial z^2_1} + \frac{\partial C}{\partial a_2^3} \frac{\partial a_2^3}{\partial z_2^3} \frac{\partial z_2^3}{\partial a_1^2} \frac{\partial a_1^2}{\partial z^2_1}\\ \end{align}$$
如果有更多层，则都是在后一层计算完成的基础上进行更新。

K个训练样本

同感知器的K个同理，使用k个样本的平均误差进行更新

批随机梯度下降

因为样本的巨大，所以一次不会使用所有的样本进行训练，而是将训练样本分为比较小的部分进行分批训练。

实现

使用三层神经网络识别手写数字。

语言:C++
数据:Mnist

具体实现在我的图像处理库:zMatrix中的zml中。仅做参考