5和6是在数据回归中用的较多的损失函数
5. nn.L1Loss
功能:计算inputs与target之差的绝对值
代码:
nn.L1Loss(reduction='mean')
公式:
6. nn.MSELoss
功能:计算inputs与target之差的平方
代码:
nn.MSELoss(reduction='mean')
主要参数:reduction:计算模式,none/sum/mean
公式:
7. SmoothL1Loss
功能:平滑的L1Loss
代码:
nn.SmoothL1Loss(size_average=None,reduce=None,reduction='mean')
$$f(x)=
\begin{cases}
0& \text{x=0}\\
1& \text{x!=0}
\end{cases}$$
8. PoissonNLLLoss
功能:泊松分布的负对数似然损失函数
主要参数:
- log_input:输入是否为对数形式,决定计算公式
- full:计算所有Loss,默认为False
- eps:修正项,避免log(input)为nan
代码:
nn.PoissonNLLLoss(log_input=True,full=False,eps=1e-08,reduction='mean')
log_input = True
loss(input,target)=exp(input)-target*input
log_input = False
loss(input,target)=input-target*log(input+eps)
9. nn.KLDivLoss
功能:计算KLD(divergence),KL散度,相对熵
注意事项:需提前将输入计算log-probabilities,如通过nn.logsoftmax(),也就是输入一个概率,概率取值的区间是(0,1)。因为在公式中有Log,因此要求我们在输入中就给log函数进行运算。
主要参数:
-reduce:none,sum,mean,batchmean.
batchmean-batchsize维度求平均值
nn.KLDivLoss(reduction='mean')
10. nn.MarginRankingLoss
功能:计算两个向量之间的相似度,用于排序任务
特别说明:该方法计算两组数据之间的差异,返回一个n*n的loss的矩阵
主要参数:
- margin:边界值,x1与x2之间的差异值
- reduction:计算模式,可为none/sum/mean
y=1时,希望x1比x2大,当x1>x2时,不产生Loss
y=-1时,希望x2比x1大,当x2>x1时,不产生Loss
nn.MarginRankingLoss(margin=0.0,size_average=None,reduce=None,reduction='mean')
11. nn.MultiLabelMarginLoss
功能:多标签边界损失函数。
比如一个样本有多个类别。
距离:四分类任务,样本x属于0类和3类
标签:[0,3-1,-1],而不是[1,0,0,1]
主要参数:
- reduction:计算模式,可为none/sum/mean
nn.MultiLabelMarginLoss(size_average=None,reduce=None,reduction='mean')
分母是x的大小,是输出向量的神经元的个数。
12. nn.SoftMarginLoss
功能:计算二分类的logistic损失
主要参数
- reduction:计算模式,可为none,sum,mean
nn.SoftMarginLoss(size_average=None,reduce=None,reduction='mean')
13. nn.MultiLabelSoftMarginLoss
功能:SoftMarginLoss多标签版本
主要参数:这里的标签是01。
- weight:各类别的Loss设置权值
- reduction:计算模式,可为none,sum,mean
nn.MultiLabelSoftMarginLoss(weight=None,size_average=None,reduce=None,reduction='mean')
14. nn.MultiMarginLoss
功能:计算多分类的折页损失
主要参数:
- p:可选1或2
- weight:各类别的Loss设置权值
- margin:边界值
- reduction:计算模式,可为none,sum,mean
nn.MultiMarginLoss(p=1,margin=1.0,weight=None,size_average=None,reduce=None,reduction='mean')
15. nn.TripletMarginLoss
功能:计算三元组损失,人脸验证中常用
主要参数:
- p :范数的阶,默认为2
- margin:边界值
- reduction:计算模式,none,sum,mean
nn.TripletMarginLoss(margin=1.0,p=2.0,eps=1e-06,swap=False,size_average=None,reduce=None,reduction='mean')
16. nn.HingeEmbeddingLoss
功能:计算两个输入的相似性,常用于非线性embedding和半监督学习
特别注意:输入x应为两个输入之差的绝对值
主要参数:
- margin:边界值
- reduction:计算模式,可为none,sum,mean
nn.HingeEmbeddingLoss(margin=1.0,size_average=None,reduce=None,reduction='mean')
17. nn.CosineEmbeddingLoss
功能:采用余弦相似度计算两个输入的相似性
主要参数:
- margin:可取值[-1,1],推荐为[0,0.5]
- reduction:计算模式,可为none,sum,mean
nn.CosineEmbeddingLoss(margin=0.0,size_average=None,reduce=None,reduction='mean')
18. nn.CTCLoss
功能:计算CTC损失,解决时序类数据的分类
Connetionist Temporal Classification
主要参数:
- blank:blank label
- zero_infinity:无穷大的值或梯度置0
- reduction:计算模式,可为none,sum,mean
torch.nn.CTCLoss(blank=0,reduction='mean',zero_infinity=False)
import torch
import torch.nn as nn
import os
import matplotlib.pyplot as plt
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"
torch.manual_seed(2)
# ===========================nn.L1Loss===========
# flag = True
flag = False
if flag:
x1_data = torch.ones((2, 2))
x1_label = torch.ones((2, 2)) * 3
loss1 = nn.L1Loss(reduction='none')
loss_data = loss1(x1_data, x1_label)
print("x1_data:{}\nx1_label:{}\nloss:{}".format(x1_data,x1_label,loss_data))
# ===========================nn.MSELoss===========
# flag = True
flag = False
if flag:
x1_data = torch.ones((2, 2))
x1_label = torch.ones((2, 2)) * -1
loss1 = nn.MSELoss(reduction='none')
loss_data = loss1(x1_data, x1_label)
print("x1_data:{}\nx1_label:{}\nloss:{}".format(x1_data,x1_label,loss_data))
# ===========================nn.SmoothLoss===========
# flag = True
flag = False
if flag:
x1_data = torch.linspace(-3, 3, steps=1000)
x1_label = torch.zeros_like(x1_data)
lossmoth1 = nn.SmoothL1Loss(reduction='none')
loss_datamoth1 = lossmoth1(x1_data, x1_label)
loss1 = nn.L1Loss(reduction='none')
loss_data = loss1(x1_data, x1_label)
plt.plot(x1_data.numpy(), loss_data.numpy(), label='L1Loss')
plt.plot(x1_data.numpy(), loss_datamoth1.numpy(), label='SmoothL1Loss')
plt.xlabel('xi-yi')
plt.ylabel('loss value')
plt.legend()
plt.grid()
plt.show()
# ===========================nn.PoissonNLLLoss===========
# flag = True
flag = False
if flag:
x1_data = torch.randn((2, 2))
x1_label = torch.randn((2, 2))*2+0.6
loss1 = nn.PoissonNLLLoss(log_input=False, reduction='none',eps=1e-02)
loss_data = loss1(x1_data, x1_label)
print(x1_data)
print(x1_label)
print(loss_data)
# computebyhan = torch.exp(x1_data)-x1_label*x1_data
# print(computebyhan)
computebyhan1 = x1_data - x1_label*torch.log(x1_data+1e-02)
print(computebyhan1)
# ===========================nn.KLDivLoss===========
# flag = True
flag = False
if flag:
x1_data = torch.tensor([[0.5, 0.4, 0.1], [0.2, 0.4, 0.4], [0.1, 0.5, 0.4], [0.1, 0.8, 0.1]])
x1_datalog = torch.log(x1_data)
x1_label = torch.tensor([[0.7, 0.2, 0.1], [0.1, 0.4, 0.5], [0.4, 0.2, 0.4], [0.3, 0.4, 0.3]],dtype=torch.float)
lossd = nn.KLDivLoss(reduction='none')
loss_data = lossd(x1_data, x1_label)
print(loss_data)
ln = x1_label*(torch.log(x1_label)-x1_data)
print(ln)
# ===========================nn.MarginRankingLoss===========
# flag = True
flag = False
if flag:
x1 = torch.tensor([[1], [2], [3]], dtype=torch.float)
x2 = torch.tensor([[2], [2], [2]], dtype=torch.float)
labeld = torch.tensor([1, 1, -1], dtype=torch.float)
loss_f = nn.MarginRankingLoss(margin=0, reduction='none')
loss = loss_f(x1, x2, labeld)
print(loss)
# ===========================nn.MultiLabelMarginLoss===========
# flag = True
flag = False
if flag:
x = torch.tensor([[0.4, 02., 0.4, 0.5]])
y = torch.tensor([[0, 3, -1, -1]], dtype=torch.long)
lossf = nn.MultiLabelMarginLoss(reduction='none')
loss = lossf(x, y)
print(loss)
x = x[0]
item1 = (1 - (x[0]-x[1]))+(1-(x[0]-x[2]))
item2 = (1 - (x[3]-x[1]))+(1-(x[3]-x[2]))
loss2 = (item1+item2)/4
print(loss2)
# ===========================nn.SoftMarginLoss===========
# flag = True
flag = False
if flag:
x = torch.tensor([[0.4, 0.6], [0.8, 0.2]])
y = torch.tensor([[-1, 1], [1, -1]], dtype=torch.long)
lossf = nn.SoftMarginLoss(reduction='none')
loss = lossf(x, y)
print(loss)
idx = 0
itemx = x[idx, idx]
itemy = y[idx, idx]
loss = torch.log(1+torch.exp(-itemy*itemx))
print(loss)
# ===========================nn.MultiLabelSoftMarginLoss===========
# flag = True
flag = False
if flag:
x = torch.tensor([[0.4, 0.6, 0.8, 0.2]])
y = torch.tensor([[0, 1, 1, 0]], dtype=torch.long)
lossf = nn.MultiLabelSoftMarginLoss(reduction='none')
loss = lossf(x, y)
print(loss)
sum=0
for i in range(4):
if(y[0,i]==1):
sum+=y[0, i]*torch.log((1+torch.exp(-x[0,i]))**(-1))
else:
sum+=torch.log(torch.exp(-x[0,i])/(1+torch.exp(-x[0,i])))
sum=sum/4
print(sum)
# ===========================nn.MultiMarginLoss===========
# ---------------------------------------------- 14 Multi Margin Loss -----------------------------------------
flag = 0
# flag = 1
if flag:
x = torch.tensor([[0.1, 0.2, 0.7], [0.2, 0.5, 0.3]])
y = torch.tensor([1, 2], dtype=torch.long)
loss_f = nn.MultiMarginLoss(reduction='none')
loss = loss_f(x, y)
print("Multi Margin Loss: ", loss)
# --------------------------------- compute by hand
flag = 0
# flag = 1
if flag:
x = x[0]
margin = 1
i_0 = margin - (x[1] - x[0])
# i_1 = margin - (x[1] - x[1])
i_2 = margin - (x[1] - x[2])
loss_h = (i_0 + i_2) / x.shape[0]
print(loss_h)
# ---------------------------------------------- 15 Triplet Margin Loss -----------------------------------------
# flag = 0
flag = 1
if flag:
anchor = torch.tensor([[1.]])
pos = torch.tensor([[2.]])
neg = torch.tensor([[0.5]])
lossd = nn.TripletMarginLoss(margin=1, p=1)
loss = lossd(anchor, pos, neg)
print(loss)
ap = abs(pos-anchor)
an = abs(anchor-neg)
l = ap-an+1
print(l)
# ---------------------------------------------- 16 Hinge Embedding Loss -----------------------------------------
flag = 0
# flag = 1
if flag:
inputs = torch.tensor([[1., 0.8, 0.5]])
target = torch.tensor([[1, 1, -1]])
loss_f = nn.HingeEmbeddingLoss(margin=1, reduction='none')
loss = loss_f(inputs, target)
print("Hinge Embedding Loss", loss)
# --------------------------------- compute by hand
flag = 0
# flag = 1
if flag:
margin = 1.
loss = max(0, margin - inputs.numpy()[0, 2])
print(loss)
# ---------------------------------------------- 17 Cosine Embedding Loss -----------------------------------------
flag = 0
# flag = 1
if flag:
x1 = torch.tensor([[0.3, 0.5, 0.7], [0.3, 0.5, 0.7]])
x2 = torch.tensor([[0.1, 0.3, 0.5], [0.1, 0.3, 0.5]])
target = torch.tensor([[1, -1]], dtype=torch.float)
loss_f = nn.CosineEmbeddingLoss(margin=0., reduction='none')
loss = loss_f(x1, x2, target)
print("Cosine Embedding Loss", loss)
# --------------------------------- compute by hand
flag = 0
# flag = 1
if flag:
margin = 0.
def cosine(a, b):
numerator = torch.dot(a, b)
denominator = torch.norm(a, 2) * torch.norm(b, 2)
return float(numerator/denominator)
l_1 = 1 - (cosine(x1[0], x2[0]))
l_2 = max(0, cosine(x1[0], x2[0]))
print(l_1, l_2)