MLP损失函数及反向传播算法

1.多层感知机模型框架

多层感知机正向传播（forward propagation）

$$a_1^2=f({\theta }_{10}^1{x}_0+{\theta }_{11}^1{x}_1+{\theta }_{12}^1{x}_2+{\theta }_{13}^1{x}_3)=f({\theta }^{1}x)=f(z_1^2)$$
$$a_2^2=f({\theta }_{20}^1{x}_0+{\theta }_{21}^1{x}_1+{\theta }_{22}^1{x}_2+{\theta }_{23}^1{x}_3)=f({\theta }^{2}x)=f(z_2^2)$$
$$a_3^2=f({\theta }_{30}^1{x}_0+{\theta }_{31}^1{x}_1+{\theta }_{32}^1{x}_2+{\theta }_{33}^1{x}_3)=f({\theta }^{3}x)=f(z_3^2)$$

$$y=f({\theta }_{10}^2{a}_0^2+{\theta }_{11}^2{a}_1^2+{\theta }_{12}^2{a}_2^2+{\theta }_{13}^2{a}_3^2)=f({\theta }^2{x}^2)=f(z_1^3)$$
模型求解：寻找到合适的参数θ,使总样本预测y与实际y的误差总和最小。

2.损失函数：从逻辑回归到mlp
逻辑回归损失函数（J）
$$J=\frac{1}{m}\sum _{i=1}^m{J}^{(i)}=-\frac{1}{m}[\sum _{i=1}^m(y^{i}log(h(x^i))+(1-y^{(i)})log(1-h(x^i)))]$$

多层感知器损失函数损失函数（J）：
$$J=\frac{1}{m}\sum _{i=1}^m\sum _{k=1}^K{J}^{(i)}=-\frac{1}{m}[\sum _{i=1}^m\sum _{k=1}^K(y_k^{(i)}log(h(x^i){)}_k+(1-y_k^{(i)})log(1-(h(x^i){)}_k))]$$

梯度下降法寻找参数
$$f(x)=\left\{\begin{array}{r}tem{p}_{{\theta }_j}={\theta }_j-\alpha \frac{\delta }{\delta {\theta }_j}J(\theta )\\ {\theta }_j=tem{p}_{{\theta }_j}\end{array}\right\}$$
以一个神经元为例：x=3，Y=10
$$y={\theta }_{1}x+{\theta }_0$$
初始参数：θ 0=2，θ1=2，α=0.2

$$J=|Y-y|=Y-({\theta }_{1}x+{\theta }_0)=10-8=2$$
$$\begin{gathered} {\theta }_{01}={\theta }_0-\alpha \frac{\delta J(\theta )}{\delta {\theta }_0}=2-0.2\ast （-1）=2.2 \\ {\theta }_{11}={\theta }_1-\alpha \frac{\delta J(\theta )}{\delta {\theta }_1}=2-0.2\ast （-3）=2.6 \end{gathered}$$
$$Y_1=2.6x+2.2=10$$

3.多层感知机反向传播（backpropagation）

如上图，只有三层：
$$\begin{gathered} {\delta }_j^l代表第l层递j个神经元的偏差 \\ {\delta }_1^3=a^3-Y \\ {\delta }^2=({\theta }^2{)}^T\odot {\delta }^3{f}^{\prime }(z^3) \end{gathered}$$

如果有更多的隐藏层：
$${\delta }^l=({\theta }^l{)}^T{\theta }^{(l+1)}\odot f^{\prime }(z^{l+1})$$
核心思想：从后往前依次计算每层神经元的数值偏差，然后通过梯度下降法寻找到使偏差最小的参数θ，完成模型求解。