Cross-Entropy loss
-
cross entropy loss
C E = − ∑ i C t i l o g ( s i ) CE = -\sum_i^C t_i log(s_i) CE=−i∑Ctilog(si)
其中ti和si 分别是C个类别中第i类别的groundtruth 和predict score .
通常激活函数 (Sigmoid/softmax)用于交叉熵损失之前计算 score (si), 一般记作:f(si) -
binary classification: C = 2
C E = − ∑ i = 1 C ′ = 2 t i l o g ( s i ) = − t 1 l o g ( s − 1 ) − ( 1 − t 1 ) l o g ( 1 − s 1 ) CE = -\sum_{i=1}^{C'=2} t_i log(s_i) = -t_1log(s-1) - (1-t_1)log(1-s_1) CE=−i=1∑C′=2tilog(si)=−t1log(s−1)−(1−t1)log(1−s1) -
Logistic loss 和 Multinomial Logistic loss
这两个只是一个CrossEntropyLoss的另外的称呼。
pytorch 中是 BCELoss(仅仅限于2分类)
tensorflow中是 log_loss
Categorical Cross-Entropy loss
这种损失也称为 softmax loss. 它是以softmax 激活 加上 CrossEntropyLoss.
如果我们在网络中使用这个loss,网络最后会输出C个类别的概率。用于多分类。
通常多分类的标签是 one-hot形式的,所以只有 正类Cp 保留loss, 其他的都为0.
其中Sp 就是网络输出的正类的score 。
对于softmax 损失计算梯度时,分别计算关于Sp(true label)的 梯度 和Sn(negative)的梯度:
在pytorch中, CrossEntropyLoss 仅限于多分类
tensorflow中,softmax_cross_entropy仅限于多分类。
Focal Loss
FocalLoss 是一种带权重的CrossEntropy loss。 权重是由每个样本在分类中的损失计算得来的。用于解决样本不均衡的问题。
Ranking Loss
参考:ranking_loss/
ranking loss是经常用于 孪生网络,或者Troplet网络。有很多名字:Contrastive Loss, Margin Loss, HingeLoss, Triplet loss.
Ranking loss:度量学习 metric learning
ranking loss 不像 CE or MSE(CE/MSE的目标是根据输入预测label 或者 一个值)。ranking loss的目标是预测输入的相对距离。是一个度量学习任务。
ranking loss在训练数据上非常灵活,我们只需要得到一个相似度得分,可以是二分类模式的 相似/不相似。
使用ranking loss函数时,一般是从输入数据里面提取2个或者3个数据 然后获取他们的embedding, 然后定义一种度量函数 测量两者之间的相似度,比如欧式距离,余弦相似度。
最后,我们训练一个特征提取器 用于产生输入的表征向量,使得相似的距离更近,不想死的更远。我们只关心的是向量的距离。
ranking 损失虽然名字各不相同,但是公式都很简单并且没啥差别。下面我们讨论2种类型的损失:pairs of training data points, triplets of training daat points. 这两种都是用来比较训练数据表征的距离。
- pairwise ranking loss
- Triplet Ranking loss
Rnaking Losses的其他称呼:
- ranking loss
- margin loss
- contrastive loss
- triplet loss
- hinge loss (max-margin objective)
在pytorch中:
- consineEmbeddingLoss: pairwise ranking loss 使用cosine distance作为距离度量。
- MarginRankingLoss
- TripletMarginLoss: 使用的是欧式距离。
在tensorflow中: - constrastive_loss: pairwise ranking loss
- triplet_semihard_loss: triplet loss with semi-hard negative mining.
Others
- RankCrossEntropyLoss: 排序交叉熵损失
class RankCrossEntropyLoss(nn.Module):
"""Creates a criterion that measures rank cross entropy loss."""
__constants__ = ['num_neg']
def __init__(self, num_neg: int = 1):
"""
:class:`RankCrossEntropyLoss` constructor.
:param num_neg: Number of negative instances in hinge loss.
"""
super().__init__()
self.num_neg = num_neg
def forward(self, y_pred: torch.Tensor, y_true: torch.Tensor):
"""
Calculate rank cross entropy loss.
:param y_pred: Predicted result.
:param y_true: Label.
:return: Rank cross loss.
"""
logits = y_pred[::(self.num_neg + 1), :] # 排序结果的前 num_neg + 1个结果
labels = y_true[::(self.num_neg + 1), :] # 真实label的前 num_neg + 1个结果
for neg_idx in range(self.num_neg):
neg_logits = y_pred[(neg_idx + 1)::(self.num_neg + 1), :]
neg_labels = y_true[(neg_idx + 1)::(self.num_neg + 1), :]
logits = torch.cat((logits, neg_logits), dim=-1)
labels = torch.cat((labels, neg_labels), dim=-1)
return -torch.mean(
torch.sum(
labels * torch.log(F.softmax(logits, dim=-1) + torch.finfo(float).eps),
dim=-1
)
)
@property
def num_neg(self):
"""`num_neg` getter."""
return self._num_neg
@num_neg.setter
def num_neg(self, value):
"""`num_neg` setter."""
self._num_neg = value
- 合页损失
class RankHingeLoss(nn.Module):
"""
Creates a criterion that measures rank hinge loss.
Given inputs :math:`x1`, :math:`x2`, two 1D mini-batch `Tensors`,
and a label 1D mini-batch tensor :math:`y` (containing 1 or -1).
If :math:`y = 1` then it assumed the first input should be ranked
higher (have a larger value) than the second input, and vice-versa
for :math:`y = -1`.
The loss function for each sample in the mini-batch is:
.. math::
loss_{x, y} = max(0, -y * (x1 - x2) + margin)
"""
__constants__ = ['num_neg', 'margin', 'reduction']
def __init__(self, num_neg: int = 1, margin: float = 1.,
reduction: str = 'mean'):
"""
:class:`RankHingeLoss` constructor.
:param num_neg: Number of negative instances in hinge loss.
:param margin: Margin between positive and negative scores.
Float. Has a default value of :math:`0`.
:param reduction: String. Specifies the reduction to apply to
the output: ``'none'`` | ``'mean'`` | ``'sum'``.
``'none'``: no reduction will be applied,
``'mean'``: the sum of the output will be divided by the
number of elements in the output,
``'sum'``: the output will be summed.
"""
super().__init__()
self.num_neg = num_neg
self.margin = margin
self.reduction = reduction
def forward(self, y_pred: torch.Tensor, y_true: torch.Tensor):
"""
Calculate rank hinge loss.
:param y_pred: Predicted result.
:param y_true: Label.
:return: Hinge loss computed by user-defined margin.
"""
y_pos = y_pred[::(self.num_neg + 1), :]
y_neg = []
for neg_idx in range(self.num_neg):
neg = y_pred[(neg_idx + 1)::(self.num_neg + 1), :]
y_neg.append(neg)
y_neg = torch.cat(y_neg, dim=-1)
y_neg = torch.mean(y_neg, dim=-1, keepdim=True)
y_true = torch.ones_like(y_pos)
return F.margin_ranking_loss(
y_pos, y_neg, y_true,
margin=self.margin,
reduction=self.reduction
)
@property
def num_neg(self):
"""`num_neg` getter."""
return self._num_neg
@num_neg.setter
def num_neg(self, value):
"""`num_neg` setter."""
self._num_neg = value
@property
def margin(self):
"""`margin` getter."""
return self._margin
@margin.setter
def margin(self, value):
"""`margin` setter."""
self._margin = value