关于softmax损失函数的推导

转载自：http://blog.csdn.net/xizero00

关于softmax损失函数的推导

某人问我softamx损失函数的推导,索性就写一下. 
定义softmax损失函数的输入为XN×CXN×C和YN×CYN×C, 
其中N代表输入的数据的个数,C代表类别的个数.X指的是神经网络的输出,Y代表的是0-1矩阵,即如果第i个样本的类别为j那么yij=1yij=1, 那么第i行的其余列的值就都为0. 这里我们用1{j=y(i)}1{j=y(i)}来表示. 
这里的softmax分类器是接在了神经网络的后面的.我们把softmax分类器看成神经网络的最后一层(请注意我的前提条件!).那么在对神经网络进行优化的时候就需要求出其关于输入X的偏导.

softmax classifier的损失函数如下: 
loss(X,Y)=−1N∑i∑j1{j=y(i)}log(pi,j)loss(X,Y)=−1N∑i∑j1{j=y(i)}log(pi,j) 
其中pi,j=exp(xi,j)∑jexp(xi,j)pi,j=exp(xi,j)∑jexp(xi,j),其含义为第i个输入的类别为j的概率为pi,jpi,j 
我们关于xi,jxi,j求偏导 
首先进行拆分 
∑i∑j1{j=y(i)}log(pi,j)=∑i[1{j=y(i)}log(pi,j)+∑c≠j1{c=y(i)}log(pi,c)]∑i∑j1{j=y(i)}log(pi,j)=∑i[1{j=y(i)}log(pi,j)+∑c≠j1{c=y(i)}log(pi,c)] 
那么损失函数为 
loss(X,Y)=−1N[∑i[1{j=y(i)}log(pi,j)+∑c≠j1{c=y(i)}log(pi,c)]]loss(X,Y)=−1N[∑i[1{j=y(i)}log(pi,j)+∑c≠j1{c=y(i)}log(pi,c)]] 
接下来进行求导 
∂loss∂xi,j=−1N[1{j=y(i)}1pi,j∂pi,j∂xi,j+∑c≠j1{c=y(i)}1pi,c∂pi,c∂xi,j]∂loss∂xi,j=−1N[1{j=y(i)}1pi,j∂pi,j∂xi,j+∑c≠j1{c=y(i)}1pi,c∂pi,c∂xi,j]

接下来我们求∂pi,j∂xi,j∂pi,j∂xi,j 
pi,j=exp(xi,j)∑jexp(xi,j)pi,j=exp(xi,j)∑jexp(xi,j) 
∂pi,j∂xi,j=exp(xi,j)∑jexp(xi,j))−exp(xi,j)exp(xi,j))[∑jexp(xi,j)]2=exp(xi,j)∑jexp(xi,j)∑jexp(xi,j)−exp(xi,j)∑jexp(xi,j)=pi,j(1−pi,j)∂pi,j∂xi,j=exp(xi,j)∑jexp(xi,j))−exp(xi,j)exp(xi,j))[∑jexp(xi,j)]2=exp(xi,j)∑jexp(xi,j)∑jexp(xi,j)−exp(xi,j)∑jexp(xi,j)=pi,j(1−pi,j) 
接下来我们求∂pi,c∂xi,j∂pi,c∂xi,j 
pi,c=exp(xi,c)∑jexp(xi,j)pi,c=exp(xi,c)∑jexp(xi,j), 
∂pi,c∂xi,j=−exp(xi,c)exp(xi,j)[∑jexp(xi,j)]2=exp(xi,c)∑jexp(xi,j)−exp(xi,j)∑jexp(xi,j)=pi,c(−pi,j)∂pi,c∂xi,j=−exp(xi,c)exp(xi,j)[∑jexp(xi,j)]2=exp(xi,c)∑jexp(xi,j)−exp(xi,j)∑jexp(xi,j)=pi,c(−pi,j)

那么就可以将上述结果带入可得 
∂loss∂xi,j=−1N[1{j=y(i)}1pi,j∂pi,j∂xi,j+∑c≠j1{c=y(i)}1pi,c∂pi,c∂xi,j]∂loss∂xi,j=−1N[1{j=y(i)}1pi,j∂pi,j∂xi,j+∑c≠j1{c=y(i)}1pi,c∂pi,c∂xi,j] 
=−1N[1{j=y(i)}1pi,jpi,j(1−pi,j)+∑c≠j1{c=y(i)}1pi,cpi,c(−pi,j)]=−1N[1{j=y(i)}1pi,jpi,j(1−pi,j)+∑c≠j1{c=y(i)}1pi,cpi,c(−pi,j)] 
=−1N[1{j=y(i)}(1−pi,j)+∑c≠j1{c=y(i)}(−pi,j)]=−1N[1{j=y(i)}(1−pi,j)+∑c≠j1{c=y(i)}(−pi,j)] 
=−1N[1{j=y(i)}−1{j=y(i)}pi,j−∑c≠j1{c=y(i)}pi,j]=−1N[1{j=y(i)}−1{j=y(i)}pi,j−∑c≠j1{c=y(i)}pi,j] 
=−1N[1{j=y(i)}−∑j1{j=y(i)}pi,j]=−1N[1{j=y(i)}−∑j1{j=y(i)}pi,j] 
=−1N[1{j=y(i)}−pi,j]=−1N[1{j=y(i)}−pi,j] 
=1N[pi,j−1{j=y(i)}]=1N[pi,j−1{j=y(i)}] 
即为所求,代码如下: