激活函数汇总，包含公式、求导过程以及numpy实现，2024年最新Python开发面试技能介绍

selu ⁡ ( x ) = λ { x  if  ( x > 0 ) α e x − α  if  ( x ⩽ 0 ) \operatorname{selu}(x)=\lambda \begin{cases}x & \text { if } (x>0) \\ \alpha e^{x}-\alpha & \text { if } (x \leqslant 0)\end{cases} selu(x)=λ{xαex−α​ if (x>0) if (x⩽0)​

1.7.2 导数

selu ⁡ ′ ( x ) = λ { 1 x > 0 α e x ⩽ 0 \operatorname{selu}^{\prime}(x)=\lambda \begin{cases}1 & x>0 \\ \alpha e^{x} & \leqslant 0\end{cases} selu′(x)=λ{1αex​x>0⩽0​

1.7.3 代码实现

import numpy as np

class SELU():

Reference : https://arxiv.org/abs/1706.02515,

https://github.com/bioinf-jku/SNNs/blob/master/SelfNormalizingNetworks_MLP_MNIST.ipynb

def init(self):

self.alpha = 1.6732632423543772848170429916717

self.scale = 1.0507009873554804934193349852946

def call(self, x):

return self.scale * np.where(x >= 0.0, x, self.alpha*(np.exp(x)-1))

def gradient(self, x):

return self.scale * np.where(x >= 0.0, 1, self.alpha * np.exp(x))

1.8 softplus

---

1.81 函数

Softplus ⁡ ( x ) = log ⁡ ( 1 + e x ) \operatorname{Softplus}(x)=\log \left(1+e^{x}\right) Softplus(x)=log(1+ex)

1.8.2 导数

log默认的底数是 e e e

f ′ ( x ) = e x ( 1 + e x ) ln ⁡ e = 1 1 + e − x = σ ( x ) f^{{\prime}(x)=\frac{e}{x}}{(1+e^{x})\ln e}=\frac{1}{1+e^{-x}}=\sigma(x) f′(x)=(1+ex)lneex​=1+e−x1​=σ(x)

1.8.3 代码实现

import numpy as np

class SoftPlus():

def call(self, x):

return np.log(1 + np.exp(x))

def gradient(self, x):

return 1 / (1 + np.exp(-x))

1.9 Swish

====================================================================

1.9.1 函数

f ( x ) = x ⋅ sigmoid ⁡ ( β x ) f(x)=x \cdot \operatorname{sigmoid}(\beta x) f(x)=x⋅sigmoid(βx)

1.9.2 导数

f ′ ( x ) = σ ( β x