BP推导

图

符号
$x_{ij}$:第j行数据，第i个神经元输入

$w_{ij}^l$：第l层的第i个神经元的到第l+1层的第j个神经元的权重

$b_l$：第l层的偏置bias

$z_i^l$：第l层第i个神经元的的输入

$a_i^l​$：第I层第i个神经元的输出

$\sigma()$：激活函数

$\delta_i^l$：第I层第i个神经元的的误差

推导

向前传播

第一层

$a_{1}^{(1)}=x_{1j}$

$a_{2}^{(1)}=x_{2j}$

第二层

$z_{1}^{(2)}=a_1^{(1)}w_1^{(1)}+a_2^{(1)}w_3^{(1)}+b_1=x_{1j}w_1^{(1)}+x_{2j}w_3^{(1)}+b_1$

$z_{2}^{(2)}=a_1^{(1)}w_2^{(1)}+a_2^{(1)}w_4^{(1)}+b_1=x_{1j}w_2^{(1)}+x_{2j}w_4^{(1)}+b_1$

$a_1^{(2)}=\sigma(z_1^{(2)})=\sigma(x_{1j}w_1^{(1)}+x_{2j}w_3^{(1)}+b_1)$

$a_2^{(2)}=\sigma(z_{2}^{(2)})=\sigma(x_{1j}w_2^{(1)}+x_{2j}w_4^{(1)}+b_1)$

第三层

$z_1^{(3)}=a_1^{(2)}w_1^{(2)}+a_2^{(2)}w_3^{(2)}+b_2=\sigma(x_{1j}w_1^{(1)}+x_{2j}w_3^{(1)}+b_1)w_1^{(2)}+\sigma(x_{1j}w_2^{(1)}+x_{2j}w_4^{(1)}+b_1)w_3^{(2)}+b_2$

$z_2^{(3)}=a_1^{(2)}w_2^{(2)}+a_2^{(2)}w_4^{(2)}+b_2=\sigma(x_{1j}w_1^{(1)}+x_{2j}w_3^{(1)}+b_1)w_2^{(2)}+\sigma(x_{1j}w_2^{(1)}+x_{2j}w_4^{(1)}+b_1)w_4^{(2)}+b_2$

$a_1^{(3)}=\sigma(z_1^{(3)})=\sigma(\sigma(x_{1j}w_1^{(1)}+x_{2j}w_3^{(1)}+b_1)w_1^{(2)}+\sigma(x_{1j}w_2^{(1)}+x_{2j}w_4^{(1)}+b_1)w_3^{(2)}+b_2)$

$a_2^{(3)}=\sigma(z_2^{(3)})=\sigma(\sigma(x_{1j}w_1^{(1)}+x_{2j}w_3^{(1)}+b_1)w_2^{(2)}+\sigma(x_{1j}w_2^{(1)}+x_{2j}w_4^{(1)}+b_1)w_4^{(2)}+b_2)$

反向传播

对每一层的残差求导，求梯度，更新权重。

w=w-\alpha {{\partial E}\over \partial w}

对第三层-第二层权重更新

$E_{toatal}=E_1^{(3)}+E_2^{(3)}$=${1\over 2}(y-a_1^{(3)})^2+{1\over2}(y-a_2^{(3)})^2={1\over 2}(y-[\sigma(z_1^{3})]])^2+{1\over2}(y-[\sigma(z_2^{(3)})])^2$

对$w_1^{2}$取值更新，也就是先对$w_1^{2}$求导,通过链式法则得到

${{\partial E_{total}}\over \partial w_1^{(2)}} = {{\partial[{1\over 2}(y-a_1^{(3)})^2+{1\over2}(y-a_2^{(3)})^2]}\over{\partial a_1^{(3)}}}*{\partial[{\sigma(z_1^{(3)})]}\over{\partial z_1^{(3)}}}*{\partial ({a_1^{(2)}w_1^{(2)}+a_2^{(2)}w_3^{(2)}+b_2)}\over{ \partial w_1^{(2)}}}$

$=(a_1^{(3)}-y)*\sigma(z_1^{(3)})'*a_1^{(2)}$

对第二层-第一层权重更新

${{\partial E_{total}}\over{\partial w_1^{(1)}}}={{\partial E_{total}}\over{\partial a_1^{(2)}}}*{{\partial \sigma(z_1^{(2)})}\over{\partial z_1^{(2)}}}*{{\partial(a_1^{(1)}w_1^{(1)}+a_2^{(1)}w_3^{(1)}+b_1)}\over{\partial w_1^{(1)}}}$

${{\partial E_{total}}\over{\partial a_1^{(2)}}}={{E_1^{(3)}}\over{\partial a_1^{(2)}}}+{{E_1^{(3)}}\over{\partial a_1^{(2)}}}= {{\partial[{1\over 2}(y-a_1^{(3)})^2]}\over{\partial a_1^{(3)}}}*{\partial[{\sigma(z_1^{(3)})]}\over{\partial z_1^{(3)}}}*{\partial ({a_1^{(2)}w_1^{(2)}+a_2^{(2)}w_3^{(2)}+b_2)}\over{ \partial a_1^{(2)}}}+ {{\partial[{1\over2}(y-a_2^{(3)})^2]}\over{\partial a_2^{(3)}}}*{\partial[{\sigma(z_2^{(3)})]}\over{\partial z_2^{(3)}}}*{\partial ({a_1^{(2)}w_2^{(2)}+a_2^{(2)}w_4^{(2)}+b_2)}\over{ \partial a_1^{(2)}}}$

$=(a_1^{(3)}-y)*\sigma(z_1^{(3)})'*w_1^{(2)}+(a_2^{(3)}-y)*\sigma(z_2^{(3)})'*w_2^{(2)}$

${{\partial E_{total}}\over{\partial w_1^{(1)}}}={{\partial E_{total}}\over{\partial a_1^{(2)}}}*{{\partial \sigma(z_1^{(2)})}\over{\partial z_1^{(2)}}}*{{\partial(a_1^{(1)}w_1^{(1)}+a_2^{(1)}w_3^{(1)}+b_1)}\over{\partial w_1^{(1)}}}$

$=[(a_1^{(3)}-y)*\sigma(z_1^{(3)})'*w_1^{(2)}+(a_2^{(3)}-y)*\sigma(z_2^{(3)})'*w_2^{(2)}]*\sigma(z_1^{(2)})'a_1^{(1)}$