断点续训练
保存断点
checkpoint_interval = 5
MAX_EPOCH = 10
xxx
xxx
start_epoch = -1
for epoch in range(start_epoch+1, MAX_EPOCH):
xxxxx
xxxx
xxxx
if (epoch+1) % checkpoint_interval == 0:
checkpoint = {"model_state_dict": net.state_dict(),
"optimizer_state_dict": optimizer.state_dict(),
"epoch": epoch}
path_checkpoint = "./checkpoint_{}_epoch.pkl".format(epoch)
torch.save(checkpoint, path_checkpoint)
加载断点
path_checkpoint = "./checkpoint_4_epoch.pkl"
checkpoint = torch.load(path_checkpoint)
net.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
start_epoch = checkpoint['epoch']
scheduler.last_epoch = start_epoch
finetune
步骤
1.加载模型
2.加载模型预训练参数
3.修改输出层
# 1.加载模型
model=xxxxx
# 2.加载模型参数
state_dict_load = torch.load('模型参数地址')
model.load_state_dict(state_dict_load
# 3.修改输出层
outlayer_input_d = model.outlayer.in_features
model.outlayer = nn.Linear(outlayer_input_d , classes)
训练方法
1.冻结预训练层
先冻结训练的层,再替换输出层
for param in model.parameters():
param.requires_grad = False
2. 根据优化器可分组的特点,给预训练的模型较小的学习率
outlayer_params_id = list(map(id, model.outlayer.parameters())) # 返回的是parameters的 内存地址
base_params = filter(lambda p: id(p) not in outlayer_params_id, model.parameters())
optimizer = optim.SGD([
{'params': base_params, 'lr': LR*0.1}, # 0
{'params': model.outlayer.parameters(), 'lr': LR}], momentum=0.9)
GPU并行
方法
1. torch.cuda.device_count():计算当前可见可用gpu数
2. torch.cuda.get_device_name():获取gpu名称
3. torch.cuda.manual_seed():为当前gpu设置随机种子
4. torch.cuda.manual_seed_all():为所有可见可用gpu设置随机种子
5. os.environ.setdefault("CUDA_VISIBLE_DEVICES", "1, 3") :设置可用的GPU
假设有4个真实存在的GPU(0,1,2,3)
设置可用的GPU为物理GPU(1,3):os.environ.setdefault("CUDA_VISIBLE_DEVICES", "1, 3")
在python中逻辑GPU为GPU0,GPU1
torch.nn.DataParallel
torch.nn.DataParallel(module, device_ids=None, output_device=None, dim=0)
module: 需要包装分发的模型
device_ids : 可分发的gpu,默认分发到所有可见可用gpu
output_device: 结果输出设备
假设batch_size=16,有4个GPU,则每个GPU上的batch_size=4
GPU剩余内存
def get_gpu_memory():
import os
os.system('nvidia-smi -q -d Memory | grep -A4 GPU | grep Free > tmp.txt')
memory_gpu = [int(x.split()[2]) for x in open('tmp.txt', 'r').readlines()]
os.system('rm tmp.txt')
return memory_gpu
'nvidia-smi:英伟达的显卡信息
-q:查询
-d:查询内容
grep:搜索
-A4:显示当前行及后面4行
grep Free:查询剩余内存
> tmp.txt:输入到txt文件中
rm tmp.txt:删除文件
gpu_memory = get_gpu_memory()
gpu_list = np.argsort(gpu_memory)[::-1]
gpu_list_str = ','.join(map(str, gpu_list))
os.environ.setdefault("CUDA_VISIBLE_DEVICES", gpu_list_str)
print("\ngpu free memory: {}".format(gpu_memory))
print("CUDA_VISIBLE_DEVICES :{}".format(os.environ["CUDA_VISIBLE_DEVICES"]))
报错
1.加载cuda训练的模型时显示cuda不可用,只能使用cpu
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU -only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
解决方式:torch.load(path_state_dict, map_location="cpu")
2.加载cuda并行训练的模型时,模型的key不匹配
RuntimeError: Error(s) in loading state_dict for FooNet: Missing key(s) in state_dict: "linears.0.weight", "linears.1.weight", "linears.2.weight". Unexpected key(s) in state_dict: "module.linears.0.weight", "module.linears.1.weight", "module.linears.2.weight".
一种方式是改模型的key,但是太麻烦了
另一种就是把state_dict的key改一下
from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict_load.items ():
namekey = k[7:] if k.startswith('module.') else k
new_state_dict[namekey] = v
细节:tensor.to(device)不是inplace操作,不对当前的tensor进行改变需要赋予新的变量
Moudule.to(device)是inplace操作