分类问题中的交叉熵损失和均方损失

分类问题的损失函数

1. 交叉熵损失（Cross Entropy Loss）：假设误差是二值分布，可以视为预测概率分布和真实概率分布的相似程度。在分类问题中有良好的应用
   

   $$CE(\theta) = -\displaystyle\sum_{i=1}^{n}{y_i*\log({\hat{y}_{i}})}$$
   其中$y_i$表示真实概率分布，只有$y_k=1$，而$y_j=0$，$j\neq k$。$\hat{y}_i$表示预测概率分布，设它的输入为$\theta_i$，则$\hat{y}_i=softmax(\theta_i)=\displaystyle\frac{\exp(\theta_i)}{\sum_{j}{\exp(\theta_j)}}$

2. 均方损失（Mean Square Loss）：假设误差是正态分布，适用于线性的输出，特点是对于与真实结果差别越大，则惩罚力度越大，这并不适用于分类问题。
   

   $$MS(\theta)=\frac{1}{n}\sum_{i=1}^{n}{(y_i-\hat{y}_{i})^2}$$
   其中$\hat{y}_i=W\cdot \theta_i$表示$\theta_i$的线性输出。

---

交叉熵的求导过程

由于只有$y_k=1$，而$y_j=0$，$j\neq k$，则有

$$\begin{eqnarray} CE(\theta) &=& -\sum_{i=1}^{n}{y_i*\log({\hat{y}_{i}})}\\ &=& -\log({\hat{y}_{k}})\\ &=& - \log(\frac{\exp(\theta_k)}{\sum\nolimits_{j}{\exp(\theta_j)}})\\ &=& - \theta_k + \log(\sum\nolimits_{j}{\exp(\theta_j)}) \end{eqnarray}$$
故
$$\begin{eqnarray} \frac{\partial{CE(\theta)}}{\partial{\theta}} = - \frac{\partial{\theta_k}}{\partial{\theta}} + \frac{\partial}{\partial{\theta}} \log(\sum\nolimits_{j}{\exp(\theta_j))} \end{eqnarray}$$
而根据 $\displaystyle\frac{\partial{\theta_k}}{\partial{\theta_k}} = 1$， $\displaystyle\frac{\partial{\theta_k}}{\partial{\theta_q}} = 0$， $q\neq k$，得到
$$\begin{eqnarray} \frac{\partial{\theta_k}}{\partial{\theta}} &=& \left[\begin{array}{c}0\\\vdots\\1\\\vdots\\0\end{array}\right] &=& y\\ \frac{\partial}{\partial{\theta_i}} \log(\sum\nolimits_{j}{\exp(\theta_j))} &=& \frac{\exp(\theta_i)}{\sum\nolimits_{j}{\exp(\theta_j)}} &=& \hat{y}_{i} \end{eqnarray}$$
最终得到
$$\frac{\partial{CE}}{\partial{\theta}} = \frac{\partial}{\partial{\theta}} \log(\sum\nolimits_{j}{\exp(\theta_j))} - \frac{\partial{\theta_k}}{\partial{\theta}} = \hat{y}-y$$

• 参考链接
• Why You Should Use Cross-Entropy Error Instead Of Classification Error Or Mean Squared Error For Neural Network Classifier Training
• What’s the difference between cross-entropy and mean-squared-error loss functions?
• Why is the Cross Entropy method preferred over Mean Squared Error?