理解Cross-Entropy Loss, Binary Cross-Entropy Loss, Softmax Loss, Logistic Loss, Focal Loss

这篇是自己写的

torch中的NLLLoss与CrossEntropyLoss_yumao1的专栏-CSDN博客nullhttps://blog.csdn.net/qq_22210253/article/details/852299881. 当label为hardlabel的时候CrossEntropyLoss(x, label) = log_softmax(x) + NLLLoss(x, label)2. 当label为softlabel的时候应该选用CrossEntropyLosshttps://blog.csdn.net/yumao1/article/details/123076589

这篇是国外大牛的博客

Understanding Categorical Cross-Entropy Loss, Binary Cross-Entropy Loss, Softmax Loss, Logistic Loss, Focal Loss and all those confusing nameshttps://gombru.github.io/2018/05/23/cross_entropy_loss/

1.对比两篇博客发现自己理解的不足之处：

1.1上述两篇文中中讨论的所有分类loss本质上都是cross entropy loss，其它的都是不同的名字叫法而已。

2.1 sigmoid和softmax只是交叉熵loss之前针对概率的不同激活函数，两个函数都可以用于二分类、多分类、多标签分类

2. Focal loss的理解

Focal Loss was introduced by Lin et al., from Facebook, in this paper. They claim to improve one-stage object detectors using Focal Loss to train a detector they name RetinaNet. Focal loss is a Cross-Entropy Loss that weighs the contribution of each sample to the loss based in the classification error. The idea is that, if a sample is already classified correctly by the CNN, its contribution to the loss decreases. With this strategy, they claim to solve the problem of class imbalance by making the loss implicitly focus in those problematic classes.
Moreover, they also weight the contribution of each class to the lose in a more explicit class balancing. They use Sigmoid activations, so Focal loss could also be considered a Binary Cross-Entropy Loss. We define it for each binary problem as:

Where \((1 - s_i)\gamma\), with the focusing parameter \(\gamma >= 0\), is a modulating factor to reduce the influence of correctly classified samples in the loss. With \(\gamma = 0\), Focal Loss is equivalent to Binary Cross Entropy Loss.

The loss can be also defined as :

Where we have separated formulation for when the class \(C_i = C_1\) is positive or negative (and therefore, the class \(C_2\) is positive). As before, we have \(s_2 = 1 - s_1\) and \(t2 = 1 - t_1\).

如果一个数据集中存在大量的噪声样本，此时用focal loss可能带来负面效果，因为噪声可能使得某个类别表现格外差，这样带来的后果是loss将该类别权重变大，同时放大了噪声的作用。但是如果噪声是均匀分布的这话不会有这种影响。