深度学习处理概率分布中常用的函数

函数

logistic sigmoid 函数： $\sigma (x) = \dfrac {1} {1 + e ^ {-x} }$
softplus 函数： $\zeta (x) = \log (1 + e ^x)$
positive part (正部)函数 $x ^+ = \max (0, x)$
negative part (负部)函数 $x ^- = \max (0, -x)$

性质

$\sigma (x) = \dfrac {e ^ {x} } {1 + e ^ {x} }$
$\dfrac {\mathrm {d}}{\mathrm {d} x} \sigma (x) = - \dfrac {1}{(1 + e ^ {-x}) ^2} (- e ^ {-x}) = \sigma (x)[1 - \sigma (x)]$
$1 - \sigma (x) = \sigma (-x)$
$\log \sigma (x) = - \zeta (-x)$
$\dfrac {\mathrm {d}}{\mathrm {d} x} \zeta (x) = \sigma (x)$
$\forall x \in (0, 1), \sigma^{-1} (x) = \log \left ( \dfrac {x} {1 - x} \right )$
$\zeta (x) = \int _{- \infty} ^{x} \sigma(t) \mathrm {d} t$
$\zeta (x) - \zeta (-x) = x$
$x ^+ - x ^- = x$