[torch]create a new criterion(cross entropy)

本文介绍了如何在PyTorch中创建自定义的交叉熵损失函数(CrossEntropyCriterion),包括考虑权重、平均化和处理零值的细节,并提供了CPU和GPU版本的实现步骤及测试代码。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

http://blog.csdn.net/u010167269/article/details/51966427

Introduction

CrossentropyCriterion in torch

CrossEntropyCriterion and ClassNLLCriterion only support with hard targets.

For examples, in multi-class classification task,
suppose we have 4 classes: dog, cat, carrot, book.
For one image whose annotation is “cat”, the label will be P=(0,1,0,0) P = ( 0 , 1 , 0 , 0 ) .
And the output of the model is Q=(0.1,0.7,0.01,0.19) Q = ( 0.1 , 0.7 , 0.01 , 0.19 )
CrossEntropyCriterion(torch)
loss=log(Q2) l o s s = − l o g ( Q 2 )
Even if the label is soft P=(0.3,0.5,0.2,0.1) P = ( 0.3 , 0.5 , 0.2 , 0.1 ) ,
CrossEntropyCriterion(torch) won’t use other classes’ label when calculating the loss.
MyCrossEntropyCriterion
In fact, the crossentropy should be loss=ipilog(qi)=p1log(q1)p2log(q2)... l o s s = − ∑ i p i l o g ( q i ) = − p 1 l o g ( q 1 ) − p 2 l o g ( q 2 ) − . . .

One more thing:
DistKLDivCriterion” is the KL divergence(relative entropy),
(pi)log(pi/qi)=[(pi)log(p

train with base lr in the first 100 epochs # and half the lr in the last 100 epochs To train with a base learning rate for the first 100 epochs and half the learning rate for the last 100 epochs, you can use a learning rate scheduler in PyTorch. Here's an example of how you can modify the training loop in your code: import torch import torch.nn as nn import torch.optim as optim from torch.optim.lr_scheduler import MultiStepLR # Define your model, criterion, and optimizer model = YourModel() criterion = nn.CrossEntropyLoss() optimizer = optim.SGD(model.parameters(), lr=0.01) # Define the number of epochs and the milestone epochs num_epochs = 200 milestones = [100] # Create a learning rate scheduler scheduler = MultiStepLR(optimizer, milestones=milestones, gamma=0.5) # Train the model for epoch in range(num_epochs): # Train with base lr for the first 100 epochs, and half the lr for the last 100 epochs if epoch >= milestones[0]: scheduler.step() for inputs, labels in train_loader: # Forward pass outputs = model(inputs) loss = criterion(outputs, labels) # Backward pass and optimization optimizer.zero_grad() loss.backward() optimizer.step() # Perform validation or testing after each epoch with torch.no_grad(): # Validation or testing code # Print training information print(f"Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item()}, LR: {scheduler.get_last_lr()[0]}") # Save the model or perform other operations after training In this code snippet, we create a MultiStepLR scheduler and specify the milestones as [100] and gamma as 0.5. The learning rate is halved at the specified milestone epochs. Inside the training loop, we check if the current epoch is greater than or equal to the milestone epoch, and if so, we call scheduler.step() to update the learning rate. Remember to adjust the num_epochs and other hyperparameters according to your specific requirements. 翻译成中文
07-16
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值