PyTorch里NLLLoss、CrossEntropyLoss、BCELoss和BCEWithLogitsLoss之间的区别和联系

NLLLoss

examples

m = nn.LogSoftmax(dim=1)
loss = nn.NLLLoss()
# input is of size N x C = 3 x 5
input = torch.randn(3, 5, requires_grad=True)
# each element in target has to have 0 <= value < C
target = torch.tensor([1, 0, 4])
output = loss(m(input), target)
output.backward()


# 2D loss example (used, for example, with image inputs)
N, C = 5, 4
loss = nn.NLLLoss()
# input is of size N x C x height x width
data = torch.randn(N, 16, 10, 10)
conv = nn.Conv2d(16, C, (3, 3))
m = nn.LogSoftmax(dim=1)
# each element in target has to have 0 <= value < C
target = torch.empty(N, 8, 8, dtype=torch.long).random_(0, C)
output = loss(m(conv(data)), target)
output.backward()

loss

unreduced

reduced

N是batchsize

理解

来自 https://www.cnblogs.com/ZJUT-jiangnan/p/5489047.html

NLLLoss前面必须要有log_softmax。拿着前面log_softmax输出的一批logy（这里的notation和上面的不一样。上面的y相当于这里的t，这里的logy相当于上面的x），找出target对应的那个logy，给它加个负号就是loss了。

CrossEntropyLoss

This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class.

example

loss = nn.CrossEntropyLoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.empty(3, dtype=torch.long).random_(5)
output = loss(input, target)
output.backward()

loss

理解

跟上面NLLLoss里cross entropy = log_softmax + NLLLoss的搭配是一致的。

就是对每一个样本预测多个值，每个值对应一个类别，经过log_softmax后取target 类别对应的那个值。

BCELoss

example

m = nn.Sigmoid()
loss = nn.BCELoss()
input = torch.randn(3, requires_grad=True)
target = torch.empty(3).random_(2)
output = loss(m(input), target)
output.backward()

loss

理解

前面一定要加sigmoid。每个样本预测一个值，经过sigmoid后加个log优化。

BCEWithLogitsLoss

就是上面那个把sigmoid层放进去，为了数值稳定。