双层神经网络与逻辑回归正反向传播数学公式简要推导

• 逻辑回归正向传播和反向传播推导:
  
  

  $$z = w^Tx + b \tag1$$
  
  $$\begin {aligned} a&= \sigma(z) \\\ &=\frac{1}{1 + e^{-z}} \\\ \end {aligned}\tag2$$
  
  $$L(a,y) = -(y\log(a) + (1 - y)\log(1 - a)) \tag3$$
  
  $$\begin {aligned} da &= \frac{\partial L(a,y)}{\partial a} \\\ & = -y\frac{1}{a\ln{e}}-(1 - y)\frac{1}{(1 - a)\ln{e}}(-1) \\\ & = -\frac{y}{a} + \frac{1 - y}{1 - a} \\\ \end {aligned}\tag4$$
  
  $$\begin {aligned} \frac{\partial a}{\partial z} &= + (1 + e^{-z})e^{-z} \\\ & = \frac{1}{1 + e^{-z}}·\frac{e^{-z}}{1 + e^{-z}} \\\ & = 1 + e^{-z}(1 - \frac{1}{1 + e^{-z}}) \\\ & = a(1 - a) \end {aligned}\tag5$$
  
  $$\begin {aligned} dz &= \frac{\partial L(a,y)}{\partial z} \\\ & = \frac{\partial L(a,y)}{\partial a}\frac{\partial a}{\partial z} \\\ & = (-\frac{y}{a} + \frac{1 - y}{1 - a}) a(1-a) \\\ & = a - y \\\ \end {aligned}\tag6$$
  
  $$\begin {aligned} dw &= \frac{\partial L(a,y)}{\partial w} \\\ & = \frac{\partial L(a,y)}{\partial a}\frac{\partial a}{\partial z}\frac{\partial z}{\partial w} \\\ & = dz·x \\\ \end {aligned}\tag7$$
  
  $$\begin {aligned} db &= \frac{\partial L(a,y)}{\partial b} \\\ & = \frac{\partial L(a,y)}{\partial a}\frac{\partial a}{\partial z}\frac{\partial z}{\partial b} \\\ & = dz·1 \\\ & = dz \\\ \end {aligned}\tag8$$

• 双层神经网络正向传播和反向传播推导：
  双层神经网络反向传播有部分和逻辑回归相似，可以直接使用其结论
  

  $$dz^{[2]} = a^{[2]} - y \tag1$$
  $$dW^{[2]} = dz^{[2]}a^{[1]^{T}} \tag2$$
  $$db^{[2]} = dz{[2]} \tag3$$
  
  $$\begin {aligned} dz^{[1]} &= \frac{\partial L(a,y)}{\partial z^{[1]}} \\\ & = \frac{\partial L(a,y)}{\partial a}\centerdot \frac{\partial a}{\partial z} \\\ & = da\centerdot g^{[1]^{\prime}}(z^{[1]}) \\\ & = W^{[2]}dz^{[2]}\centerdot g^{[1]^{\prime}}(z^{[1]}) \\\ \end {aligned}\tag4$$
  
  $$dW^{[1]} = dz^{[1]}x^{[T]}\tag5$$
  $$db^{[1]} = dz^{[1]}\tag6$$