深度学习——激活函数sigmoid与relu的反向传播原理

一. sigmoid函数

• sigmoid函数公式
  $$\sigma (z)=\frac{1}{1+e^{-z}}\text{\hspace{1em}(1)}$$
• sigmoid函数Python实现

def sigmoid(Z):
    """
    Implements the sigmoid activation in numpy
    
    Arguments:
    Z -- numpy array of any shape
    
    Returns:
    A -- output of sigmoid(z), same shape as Z
    cache -- returns Z as well, useful during backpropagation
    """
    
    A = 1/(1+np.exp(-Z))
    cache = Z

    return A, cache

• 注：因为反向传播要用到Z，所以先将其储存在cache里

二. sigmoid函数反向传播原理

• sigmoid函数导数
  $${\sigma }^{\prime }(z)=\sigma (z)\ast (1-\sigma (z))\text{\hspace{1em}(2)}$$

• sigmoid函数反向传播原理
  在第 $l$ 层神经网络，正向传播计算公式如下：
  $$Z^{[l]}=W^{[l]}{A}^{[l-1]}+b^{[l]}\text{\hspace{1em}(3)}$$
  $$A^{[l]}=\sigma (Z^{[l]})\text{\hspace{1em}(4)}$$
  
  其中(1)为线性部分，(2)为激活部分，激活函数为sigmoid函数
  在反向传播中，计算到第 $l$ 层时，会通过后一层得到 $d{A}^{[l]}$ (即$\frac{\partial \mathcal{L}}{\partial A^{[l]}}$，其中$\mathcal{L}$为成本函数)
  当前层需要计算 $d{Z}^{[l]}$ (即$\frac{\partial \mathcal{L}}{\partial Z^{[l]}}$)，公式如下：
  $$d{Z}^{[l]}=\frac{\partial \mathcal{L}}{\partial Z^{[l]}}=\frac{\partial \mathcal{L}}{\partial A^{[l]}}\ast \frac{\partial A^{[l]}}{\partial Z^{[l]}}=dA\ast {\sigma }^{\prime }(Z^{[l]})=dA\ast \sigma (z)\ast (1-\sigma (z))\text{\hspace{1em}(5)}$$
  因此实现代码如下：

• sigmoid函数反向传播Python实现

def sigmoid_backward(dA, cache):
    """
    Implement the backward propagation for a single SIGMOID unit.
    
    Arguments:
    dA -- post-activation gradient, of any shape
    cache -- 'Z' where we store for computing backward propagation efficiently
    
    Returns:
    dZ -- Gradient of the cost with respect to Z
    """

    Z = cache

    s = 1/(1+np.exp(-Z))
    dZ = dA * s * (1-s)

    assert (dZ.shape == Z.shape)

    return dZ

三. relu函数

• relu函数公式
  $$relu(z)=\{\begin{array}{ll}z& z>0\\ 0& z\le 0\end{array}\text{\hspace{1em}(6)}$$
  等价于
  $$relu(z)=max(0,z)\text{\hspace{1em}(7)}$$
• relu函数Python实现

def relu(Z):
    """
    Implement the RELU function.
    
    Arguments:
    Z -- Output of the linear layer, of any shape
    
    Returns:
    A -- Post-activation parameter, of the same shape as Z
    cache -- a python dictionary containing "A" ; stored for computing the backward pass efficiently
    """

    A = np.maximum(0,Z)

    assert(A.shape == Z.shape)
    
    cache = Z 
    return A, cache

• 注：因为反向传播要用到Z，所以先将其储存在cache里

四. relu函数反向传播原理

• relu函数导数
  $$rel{u}^{\prime }(z)=\{\begin{array}{ll}1& z>0\\ 0& z\le 0\end{array}\text{\hspace{1em}(8)}$$

• relu函数反向传播原理
  与sigmoid同理，正向传播时，计算公式如下：
  $$Z^{[l]}=W^{[l]}{A}^{[l-1]}+b^{[l]}\text{\hspace{1em}(9)}$$
  $$A^{[l]}=relu(Z^{[l]})\text{\hspace{1em}(10)}$$
  反向传播时，$d{Z}^{[l]}$的计算公式如下：
  $$d{Z}^{[l]}=\frac{\partial \mathcal{L}}{\partial Z^{[l]}}=\frac{\partial \mathcal{L}}{\partial A^{[l]}}\ast \frac{\partial A^{[l]}}{\partial Z^{[l]}}=dA\ast rel{u}^{\prime }(Z^{[l]})=\{\begin{array}{ll}d{A}_{ij}& Z_{ij}>0\\ 0& Z_{ij}\le 0\end{array}\text{\hspace{1em}(11)}$$
  因此实现的代码如下：

• relu函数反向传播Python实现

def relu_backward(dA, cache):
    """
    Implement the backward propagation for a single RELU unit.
    
    Arguments:
    dA -- post-activation gradient, of any shape
    cache -- 'Z' where we store for computing backward propagation efficiently

    Returns:
    dZ -- Gradient of the cost with respect to Z
    """
    
    Z = cache
    dZ = np.array(dA, copy=True) # just converting dz to a correct object.
    
    # When z <= 0, you should set dz to 0 as well. 
    dZ[Z <= 0] = 0
    
    assert (dZ.shape == Z.shape)

    return dZ