selu ( x ) = λ { x if ( x > 0 ) α e x − α if ( x ⩽ 0 ) \operatorname{selu}(x)=\lambda \begin{cases}x & \text { if } (x>0) \\ \alpha e^{x}-\alpha & \text { if } (x \leqslant 0)\end{cases} selu(x)=λ{xαex−α if (x>0) if (x⩽0)
1.7.2 导数
selu ′ ( x ) = λ { 1 x > 0 α e x ⩽ 0 \operatorname{selu}^{\prime}(x)=\lambda \begin{cases}1 & x>0 \\ \alpha e^{x} & \leqslant 0\end{cases} selu′(x)=λ{1αexx>0⩽0
1.7.3 代码实现
import numpy as np
class SELU():
Reference : https://arxiv.org/abs/1706.02515,
https://github.com/bioinf-jku/SNNs/blob/master/SelfNormalizingNetworks_MLP_MNIST.ipynb
def init(self):
self.alpha = 1.6732632423543772848170429916717
self.scale = 1.0507009873554804934193349852946
def call(self, x):
return self.scale * np.where(x >= 0.0, x, self.alpha*(np.exp(x)-1))
def gradient(self, x):
return self.scale * np.where(x >= 0.0, 1, self.alpha * np.exp(x))
1.81 函数
Softplus ( x ) = log ( 1 + e x ) \operatorname{Softplus}(x)=\log \left(1+e^{x}\right) Softplus(x)=log(1+ex)
1.8.2 导数
log默认的底数是 e e e
f ′ ( x ) = e x ( 1 + e x ) ln e = 1 1 + e − x = σ ( x ) f{\prime}(x)=\frac{e{x}}{(1+e^{x})\ln e}=\frac{1}{1+e^{-x}}=\sigma(x) f′(x)=(1+ex)lneex=1+e−x1=σ(x)
1.8.3 代码实现
import numpy as np
class SoftPlus():
def call(self, x):
return np.log(1 + np.exp(x))
def gradient(self, x):
return 1 / (1 + np.exp(-x))
====================================================================
1.9.1 函数
f ( x ) = x ⋅ sigmoid ( β x ) f(x)=x \cdot \operatorname{sigmoid}(\beta x) f(x)=x⋅sigmoid(βx)