Activation function in Neural Network

最新推荐文章于 2024-05-15 10:24:36 发布

jiongjiongai

最新推荐文章于 2024-05-15 10:24:36 发布

阅读量283

点赞数

分类专栏：机器学习

本文链接：https://blog.csdn.net/phoenix198425/article/details/79762357

版权

机器学习专栏收录该内容

28 篇文章 0 订阅

订阅专栏

Logistic / Sigmoid function

$g(x) = \dfrac {1} {1 + e ^{-x}} = \dfrac {e ^{x}} {1 + e ^{x}}$
$g(-x) = \dfrac {1} {1 + e ^{x}} = \dfrac {e ^{-x}} {1 + e ^{-x}}$
$g(x) + g(-x) = 1$
$g(0) = \dfrac {1} {2}$
$\lim \limits_{x \to + \infty} g(x) = 1, \lim \limits_{x \to - \infty} g(x) = 0$
$g'(x) = - \dfrac {1} {\left (1 + e ^{-x} \right ) ^2} \cdot e ^{-x} \cdot (-1) = \dfrac {1} {\left (1 + e ^{-x} \right ) ^2} e ^{-x}$
$= \dfrac {1} {1 + e ^{-x}} \cdot \dfrac {e ^{-x}} {1 + e ^{-x}} = g(x)g( - x)= g(x) \left [1 - g(x) \right ]$
$g'(x) \gt 0, x \in \mathbb R$
$g''(x) = \left \{ g(x) \left [1 - g(x) \right ] \right \}'$
$= g'(x)\left [1 - g(x) \right ] + g(x)\left [1 - g(x) \right ]'$
$= g'(x)\left [1 - g(x) \right ] - g(x)g'(x)$
$= g'(x) \left [1 - 2 g(x) \right ]$
$= g(x) \left [1 - g(x) \right ] \left [1 - 2 g(x) \right ]$
$g''(x) \begin{cases} \lt 0, x \gt 0, \\ = 0, x = 0, \\ \gt 0, x \lt 0, \end{cases}$

tanh function

$\operatorname {tanh} (x) = 2 g(2x) - 1 = \dfrac {e ^{x} - e ^{-x}} {e ^{x} + e ^{-x}}$

$\operatorname {tanh} '(x) = 2 g'(2x) \cdot 2 = 4 g(2x) \left [ 1 - g(2x) \right ] = \left [ 1 + \operatorname {tanh} (x) \right ] \left [ 1 - \operatorname {tanh} (x) \right ]$
$= 1 - \left [ \operatorname {tanh} (x) \right ] ^2$