pytorch四种loss函数, softmax用法

pytorch四种loss函数

nn.functional.cross_entropy vs nll_loss

适用于k分类问题

labels = torch.tensor([1, 0, 2], dtype=torch.long)
logits = torch.tensor([[2.5, -0.5, 0.1],
                       [-1.1, 2.5, 0.0],
                        [1.2, 2.2, 3.1]], dtype=torch.float)
>>> torch.nn.functional.cross_entropy(logits, labels)
tensor(2.4258)
>>> torch.nn.functional.nll_loss(torch.nn.functional.log_softmax(logits, dim=1), labels)
tensor(2.4258)
>>>l=torch.nn.CrossEntropyLoss()
>>>l(logits,labels)
tensor(2.4258)

3个分类(每行三个数),每个分类都有自己的概率.
3 samples
nll_loss需要手动softmax,把三个分类的概率归一化(相加为1,因此dim=1),再取log
cross_entropy内置了log_softmax功能
logits和labels分别是FloatTensor,LongTensor

nn.CrossEntropyLoss

需要先建立一个CrossEntropyLoss层,再像cross_entropy一样代入logits和labels,不能直接用constructor代入logits和labels
否则会有一个类似Boolean value of Tensor with more than one value is ambiguous的错误,因为constructor以为输入的是boolean参数

nn.NLLLoss

输入都是size=[N]的1Dtensor，一个是log(概率)，softmax就是算概率，另一个是label

nn.KLDivLoss

输入是log(概率)和label，label也是概率

# this is the same example in wiki
P = torch.Tensor([0.36, 0.48, 0.16])
Q = torch.Tensor([0.333, 0.333, 0.333])
(P * (P / Q).log()).sum()
# tensor(0.0863)
F.kl_div(Q.log(), P, None, None, 'sum')
# tensor(0.0863)

P = torch.Tensor([[0.36, 0.48, 0.16],[0.36, 0.48, 0.16]])
Q = torch.Tensor([