1.多GPU训练记得DataLoader(dataset=dataset_train, batch_size=config['TRAIN']['BATCH'], shuffle=config['TRAIN']['SHUFFLE'], num_workers=config['TRAIN']['WORKERS'],drop_last=True)中的drop_last=True,把最后一块数据丢掉,不然最后报错。
2.如果BN在多GPU要同步,那么就要用torch.nn.SyncBatchNorm.convert_sync_batchnorm(net).to(device_ids[0]),并且要在这个代码前面,先初设化:dist.init_process_group('gloo', init_method='file:///tmp/somefile', rank=0, world_size=1)net = torch.nn.DataParallel(net, device_ids=device_ids),具体情况如下
import torch.distributed as dist dist.init_process_group('gloo', init_method='file:///tmp/somefile', rank=0, world_size=1)net = torch.nn.DataParallel(net, device_ids=device_ids) if config["TRAIN"]["DATAPARALLEL"]["syncbatchnorm"]: net = torch.nn.SyncBatchNorm.convert_sync_batchnorm(net).to(device_ids[0]) else: net = net.cuda(device=device_ids[0])