参考 torch.nn.SyncBatchNorm - 云+社区 - 腾讯云
torch.nn.
SyncBatchNorm
(num_features, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, process_group=None)[source]
Applies Batch Normalization over a N-Dimensional input (a mini-batch of [N-2]D inputs with additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.
The mean and standard-deviation are calculated per-dimension over all mini-batches of the same process groups. and