Deep learning I - III Shallow Neural Network - Derivatives of activation functions激励函数导数推导

Derivatives of activation functions 激励函数的导数

---

Derivatives of Sigmod

$$sigmod = g(z) = \frac{1}{1+e^{-z}}\tag{1}$$
$$\begin{split} g'(z) &= \frac{{\rm d}}{{\rm d}z}g(z) \\&= \frac{{\rm d}(1 + {e^{-z}})^{-1}}{{\rm d}(1 + {e^{-z}})} * \frac{{\rm d}(1 + {e^{-z}})}{{\rm d}{(-z)}} * \frac{{\rm d}{(-z)}}{{\rm d}z} \\&= -(1 + {e^{-z}})^{-2} * e^{-z} * -1 \\&= \frac{1}{(1 + e^{-z})} * \frac{e^{-z}}{(1 + e^{-z})} \\&= \frac{1}{(1 + e^{-z})} * \frac{1 + e^{-z} -1}{(1 + e^{-z})} \\&= \frac{1}{(1 + e^{-z})} *( 1- \frac{1}{(1 + e^{-z})}) \\&= g(z) * (1-g(z))\end{split}\tag{2}$$

Derivatives of tanh

$$sinh(z) = \frac{e^z - e^{-z}}{2}\tag{3}$$
$$cosh(z) = \frac{e^z + e^{-z}}{2}\tag{4}$$
$$\frac{{\rm d}}{{\rm d}z}sinh(z) = \frac{e^z + e^{-z}}{2} = cosh(z)\tag{5}$$
$$\frac{{\rm d}}{{\rm d}z}cosh(z) = \frac{e^z - e^{-z}}{2} = sinh(z)\tag{6}$$
$$tanh(z) = \frac{sinh(z)}{cosh(z)} = g(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}\tag{7}$$
$$\begin{split}g'(z) &= \frac{{\rm d}}{{\rm d}z}(sinh(z)cosh(z)^{-1}) \\&= \frac{{\rm d}}{{\rm d}z}sinh(z) * cosh(z)^{-1} + sinh(z)*\frac{{\rm d}}{{\rm d}cosh(z)}cosh(z)^{-1} * \frac{{\rm d}}{{\rm d}z}cosh(z) \\&= 1 + sinh(z)*-\frac{1}{cosh(z)^2}*sinh(z) \\&= 1 - tanh(z)^2 \end{split}\tag{8}$$

Derivatives of ReLU

$$ReLU = g(z) = max(0, z)\tag{9}$$
$$g'(z) = \begin{cases} 0 & if\ z < 0\\ 1 & if\ z \geq 0 \end{cases}\tag{10}$$
$$Leaky\ ReLU = g(z) = max(0.01z, z)\tag{11}$$
$$g'(z) = \begin{cases} 0.01 & if\ z < 0\\ 1 & if\ z \geq 0 \end{cases}\tag{12}$$