BatchNorm和LayerNorm 简单解释

对于一维的数据，使用BatchNorm1d和LayerNorm，简单的理解，BatchNorm是对所有样本的每一个维度取均值，均方差进行计算。LayerNorm是对每一个样本的所有维度取均值，均方差进行计算。

import torch
import torch.nn as nn

if __name__ == '__main__':

    input = torch.tensor([[6,6,6,6],
                          [6,6,8,4],
                          [6,8,2.,6]])
    norm1 = nn.BatchNorm1d(input.shape[1])
    norm2 = nn.LayerNorm(input.shape[1])
    print(norm1(input))
    print(norm2(input))

结果如下：

tensor([[ 0.0000, -0.7071,  0.2673,  0.7071],
        [ 0.0000, -0.7071,  1.0690, -1.4142],
        [ 0.0000,  1.4142, -1.3363,  0.7071]],
       grad_fn=<NativeBatchNormBackward>)
tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00],
        [-1.1921e-07, -1.1921e-07,  1.4142e+00, -1.4142e+00],
        [ 2.2942e-01,  1.1471e+00, -1.6059e+00,  2.2942e-01]],
       grad_fn=<NativeLayerNormBackward>)