摘要
本文给出 softmax 函数的定义, 并求解其在反向传播中的梯度
相关
配套代码, 请参考文章 :
Python 和 PyTorch 对比实现 softmax 及其反向传播
系列文章索引 :
https://blog.csdn.net/oBrightLamp/article/details/85067981
正文
1. 定义
softmax函数常用于多分类问题的输出层.
定义如下:
s i = e x i ∑ t = 1 k e x t ∑ t = 1 k e x t = e x 1 + e x 2 + e x 3 + ⋯ + e x k i = 1 , 2 , 3 , ⋯ , k s_{i} = \frac{e^{x_{i}}}{ \sum_{t = 1}^{k}e^{x_{t}}} \\ \quad \\ \sum_{t = 1}^{k}e^{x_{t}} = e^{x_{1}} + e^{x_{2}} +e^{x_{3}} + \cdots +e^{x_{k}}\\ \quad \\ i = 1, 2, 3, \cdots, k si=∑t=1kextexit=1∑kext=ex1+ex2+ex3+⋯+exki=1,2,3,⋯,k
编程实现softmax函数计算的时候, 因为存在指数运算 e x i e^{x_i} exi, 数值有可能非常大, 导致大数溢出.
一般在分式的分子和分母都乘以一个常数C, 变换成:
s i = C e x i C ∑ t = 1 k e x t = e x i + l o g C ∑ t = 1 k e x t + l o g C = e x i − m ∑ t = 1 k e x t − m m = − l o g C = m a x ( x i ) s_{i} = \frac{Ce^{x_{i}}}{ C\sum_{t = 1}^{k}e^{x_{t}}} = \frac{e^{x_{i} + logC }}{ \sum_{t = 1}^{k}e^{x_{t} + logC}} = \frac{e^{x_{i} - m }}{ \sum_{t = 1}^{k}e^{x_{t} - m}} \\ \quad \\ m = - logC = max(x_{i}) si=C∑t=1kextCexi=∑t=1kext+logCexi+logC=∑t=1kext−mexi−mm=−logC=max(xi)
C的值可自由选择, 不会影响计算结果. 这里 m 取 x i x_i xi 的最大值, 将数据集的最大值偏移至0.
2. 梯度求导
考虑一个 softmax 变换:
x = ( x 1 , x 2 , x 3 , ⋯ , x k ) s = s o f t m a x ( x ) x = (x_1, x_2, x_3, \cdots, x_k)\\ \quad\\ s = softmax(x)\\ x=(x1,x2,x3,⋯,xk)s=softmax(x)
求 s1 对 x1 的导数:
s 1 = e x 1 ∑ t = 1 k e x t = e x 1 s u m s u m = ∑ t = 1 k e x t = e x 1 + ∑ t = 2 k e x t ∂ s u m ∂ x 1 = ∂ ∑ t = 1 k e x t ∂ x 1 = e x 1 ∂ s 1 ∂ x 1 = e x 1 ⋅ s u m − e x 1 ⋅ ∂ s u m ∂ x 1 s u m 2 = e x 1 ⋅ s u m − e x 1 ⋅ e x 1 s u m 2 = s 1 − s 1 2 s_{1} = \frac{e^{x_{1}}}{ \sum_{t = 1}^{k}e^{x_{t}}} = \frac{e^{x_{1}}}{ sum} \\ \quad \\ sum = \sum_{t = 1}^{k}e^{x_{t}} = e^{x_{1}} + \sum_{t = 2}^{k}e^{x_{t}}\\ \quad \\ \frac{\partial sum}{\partial x_{1}} = \frac{\partial \sum_{t = 1}^{k}e^{x_{t}}}{\partial x_{1}} = e^{x_{1}}\\ \quad \\ \frac{\partial s_{1}}{\partial x_{1}} =\frac{e^{x_{1}} \cdot sum -e^{x_{1}}\cdot \frac{\partial sum}{\partial x_{1}}}{sum^{2}}\\ \quad\\ =\frac{e^{x_{1}} \cdot sum -e^{x_{1}} \cdot e^{x_{1}}}{sum^{2}}\\ \quad\\ = s_{1} - s_{1}^{2} \\ s1=∑t=1kextex1=sumex1sum=