激活函数汇总，包含公式、求导过程以及numpy实现，2024Python高频精选面试题讲解

推导过程如下：

f ′ = ( e i ∑ i e i ) ′ = ( e i ) × ∑ i e i − e i × e i ( ∑ i e i ) 2 = e i ∑ i e i − e i ∑ i e i × e i ∑ i e i = e i ∑ i e i ( 1 − e i ∑ i e i ) = f ( 1 − f ) \begin{aligned} f^{{\prime}&=\left(\frac{e}{i}}{\sum_{i} e^{i}}\right){\prime} & \\ &=\frac{\left(e^{i}\right) \times \sum_{i} e^{i}-e{i} \times e^{i}}{\left(\sum_{i} e^{i}\right){2}} \\ &=\frac{e^{i}}{\sum_{i} e^{i}}-\frac{e{i}}{\sum_{i} e^{i}} \times \frac{e^{i}}{\sum_{i} e^{i}} \\ &= \frac{e^{i}}{\sum_{i} e^{{i}}\left(1-\frac{e}{i}}{\sum_{i} e^{i}}\right) \\ &= f(1-f) \end{aligned} f′​=(∑i​eiei​)′=(∑i​ei)2(ei)×∑i​ei−ei×ei​=∑i​eiei​−∑i​eiei​×∑i​eiei​=∑i​eiei​(1−∑i​eiei​)=f(1−f)​

1.2.3 代码实现

import numpy as np

class Softmax():

def call(self, x):

e_x = np.exp(x - np.max(x, axis=-1, keepdims=True))

return e_x / np.sum(e_x, axis=-1, keepdims=True)

def gradient(self, x):

p = self.call(x)

return p * (1 - p)

1.3 tanh

---

1.3.1 函数

t a n h ( x ) = e x − e − x e x + e − x tanh(x)=\frac{e^{x}-e{-x}}{e^{x}+e{-x}} tanh(x)=ex+e−xex−e−x​

1.3.2 导数

求导过程：

tanh ⁡ ( x ) ′ = ( e x − e − x e x + e − x ) ′ = ( e x − e − x ) ′ ( e x + e − x ) − ( e x − e − x ) ( e x + e − x ) ′ ( e x + e − x ) 2 = ( e x + e − x ) 2 − ( e x ⋅ e − x ) 2 ( e x + e − x ) 2 = 1 − ( e x − e − x e x + e − x ) 2 = 1 − tanh ⁡ ( x ) 2 \begin{aligned} \tanh (x)^{\prime} &=\left(\frac{e^{x}-e{-x}}{e^{x}+e{-x}}\right)^{\prime} \\ &=\frac{\left(e^{x}-e{-x}\right)^{{\prime}\left(e}{x}+e^{{-x}\right)-\left(e}{x}-e^{{-x}\right)\left(e}{x}+e^{-x}\right){\prime}}{\left(e^{x}+e{-x}\right)^{2}} \\ &=\frac{\left(e^{x}+e{-x}\right)^{2}-\left(e{x} \cdot e^{-x}\right){2}}{\left(e^{x}+e{-x}\right)^{2}} \\ &=1-\left(\frac{e^{x}-e{-x}}{e^{x}+e{-x}}\right)^{2} \\ &=1-\tanh (x)^{2} \end{aligned} tanh(x)′​=(ex+e−xex−e−x​)′=(ex+e−x)2(ex−e−x)′(ex+e−x)−(ex−e−x)(ex+e−x)′​=(ex+e−x)2(ex+e−x)2−(ex⋅e−x)2​=1−(ex+e−xex−e−x​)2=1−tanh(x)2​

1.3.3 代码实现

import numpy as np

class TanH():

def call(self, x):

return 2 / (1 + np.exp(-2*x)) - 1

def gradient(self, x):

return 1 - np.power(self.call(x), 2)

1.4 relu

---

1.4.1 函数

f ( x ) = max ⁡ ( 0 , x ) f(x)=\max (0, x) f(x)=max(0,x)

1.4.2 导数