对于一个维度为NCHW(分别表示batch size, 通道数,高,宽)的特征feature tensor,
Batch Normalization:
- 对C个NHW切片张量分别归一化(即对每个特征,横跨所有samples in the batch进行归一化)
- 卷积用这个
# 看看维度如何
import torch
a = torch.ones(64,8,7,7) #feature map: batch size, 通道数,高,宽
BN = torch。nn.BatchNorm2d(8)
bn = BN(a)
print(bn.shape, BN.running_mean.shape, BN.running_var.shape, BN.weight.shape, BN.bias.shape)
#torch.Size([64, 8, 7, 7]) torch.Size([8]) torch.Size([8]) torch.Size([8]) torch.Size([8])
Layer Normalization:
- 对N个CHW切片张量分别归一化 (及对每个sample in the batch, 横跨所有特征进行归一化)
- Transformer用这个(因为时序长度长短不一)