关于softmax损失函数的推导

关于softmax损失函数的推导

某人问我softamx损失函数的推导,索性就写一下.
定义softmax损失函数的输入为$X_{N \times C}$和$Y_{N \times C}$,
其中N代表输入的数据的个数,C代表类别的个数.X指的是神经网络的输出,Y代表的是0-1矩阵,即如果第i个样本的类别为j那么$y_{ij}=1$, 那么第i行的其余列的值就都为0. 这里我们用$1\{j=y^{(i)}\}$来表示.
这里的softmax分类器是接在了神经网络的后面的.我们把softmax分类器看成神经网络的最后一层(请注意我的前提条件!).那么在对神经网络进行优化的时候就需要求出其关于输入X的偏导.

softmax classifier的损失函数如下:
$loss(X,Y) = -\frac{1}{N}\sum_{i}\sum_{j}1\{j=y^{(i)}\}log(p_{i,j})$
其中$p_{i,j} = \frac{exp(x_{i,j})}{\sum_j exp(x_{i,j})}$,其含义为第i个输入的类别为j的概率为$p_{i,j}$
我们关于$x_{i,j}$求偏导
首先进行拆分
$\sum_{i}\sum_{j}1\{j=y^{(i)}\}log(p_{i,j}) = \sum_{i} [1\{j=y^{(i)}\}log(p_{i,j})+\sum_{c\neq j}1\{c=y^{(i)}\}log(p_{i,c}) ]$
那么损失函数为
$loss(X,Y) = -\frac{1}{N}[\sum_{i} [1\{j=y^{(i)}\}log(p_{i,j})+\sum_{c\neq j}1\{c=y^{(i)}\}log(p_{i,c})]]$
接下来进行求导
$\frac{\partial loss}{\partial x_{i,j}} = -\frac{1}{N}[1\{j=y^{(i)}\}\frac{1}{p_{i,j}}\frac{\partial p_{i,j}}{\partial x_{i,j}}+\sum_{c\neq j}1\{c=y^{(i)}\}\frac{1}{p_{i,c}}\frac{\partial p_{i,c}}{\partial x_{i,j}}]$

接下来我们求$\frac{\partial p_{i,j}}{\partial x_{i,j}}$
$p_{i,j} = \frac{exp(x_{i,j})}{\sum_j exp(x_{i,j})}$
$\frac{\partial p_{i,j}}{\partial x_{i,j}} = \frac{exp(x_{i,j})\sum_{j}exp(x_{i,j})) - exp(x_{i,j}) exp(x_{i,j}))}{[\sum_{j}exp(x_{i,j})]^2} = \frac{exp(x_{i,j})}{\sum_{j}exp(x_{i,j})} \frac{ \sum_{j}exp(x_{i,j}) -exp(x_{i,j}) }{ \sum_{j} exp(x_{i,j}) } = p_{i,j}(1-p_{i,j})$
接下来我们求$\frac{\partial p_{i,c}}{\partial x_{i,j}}$
$p_{i,c} = \frac{exp(x_{i,c})}{\sum_j exp(x_{i,j})}$,
$\frac{\partial p_{i,c}}{\partial x_{i,j}} = \frac{- exp(x_{i,c}) exp(x_{i,j})}{[\sum_{j}exp(x_{i,j})]^2} = \frac{exp(x_{i,c})}{\sum_{j}exp(x_{i,j})} \frac{-exp(x_{i,j}) }{ \sum_{j} exp(x_{i,j}) } = p_{i,c}(-p_{i,j})$

那么就可以将上述结果带入可得
$\frac{\partial loss}{\partial x_{i,j}} = -\frac{1}{N}[1\{j=y^{(i)}\}\frac{1}{p_{i,j}}\frac{\partial p_{i,j}}{\partial x_{i,j}}+\sum_{c\neq j}1\{c=y^{(i)}\}\frac{1}{p_{i,c}}\frac{\partial p_{i,c}}{\partial x_{i,j}} ]$
$=-\frac{1}{N}[ 1\{j=y^{(i)}\}\frac{1}{p_{i,j}} p_{i,j}(1-p_{i,j}) + \sum_{c\neq j}1\{c=y^{(i)}\}\frac{1}{p_{i,c}} p_{i,c}(-p_{i,j}) ]$
$=-\frac{1}{N}[ 1\{j=y^{(i)}\}(1-p_{i,j}) + \sum_{c\neq j}1\{c=y^{(i)}\}(-p_{i,j}) ]$
$=-\frac{1}{N}[ 1\{j=y^{(i)}\}- 1\{j=y^{(i)}\} p_{i,j} - \sum_{c\neq j}1\{c=y^{(i)}\}p_{i,j} ]$
$=-\frac{1}{N}[ 1\{j=y^{(i)}\} - \sum_{j}1\{j=y^{(i)}\}p_{i,j} ]$
$=-\frac{1}{N}[ 1\{j=y^{(i)}\} - p_{i,j} ]$
$=\frac{1}{N}[ p_{i,j} - 1\{j=y^{(i)}\} ]$
即为所求,代码如下:

转载请注明出处：http://blog.csdn.net/xizero00