BatchNorm、LayerNorm详细过程及示例_Pytorch

一、BatchNorm

1、对batch中的每个维度进行归一化

$$y=\frac{x-E[x]}{\sqrt{Var[x]}}$$其中：$E[x]=\frac{1}{n}{\sum }_{i=1}^n{x}_i,Var[x]=\frac{1}{n}{\sum }_{i=1}^n(x_i-E[x]{)}^2$

2、pytorch示例

import torch
from torch import nn
m = nn.BatchNorm1d(3)
input = torch.tensor([[10., 2., 100.], 
                      [2., 3., 100.], 
                      [3., 40., 400.], 
                      [5., 3., 200.]], dtype=torch.float32)
print(input)
output = m(input)
print(output)

tensor([[ 10.,   2., 100.],
        [  2.,   3., 100.],
        [  3.,  40., 400.],
        [  5.,   3., 200.]])
tensor([[ 1.6222, -0.6184, -0.8165],
        [-0.9733, -0.5566, -0.8165],
        [-0.6489,  1.7315,  1.6330],
        [ 0.0000, -0.5566,  0.0000]], grad_fn=<NativeBatchNormBackward>)

二、LayerNorm

1、对一个样本的d个维度进行归一化

$$y=\frac{x-E[x]}{\sqrt{Var[x]}}$$其中：$E[x]=\frac{1}{n}{\sum }_{i=1}^n{x}_i,Var[x]=\frac{1}{n}{\sum }_{i=1}^n(x_i-E[x]{)}^2$

2、pytorch示例

import torch
from torch import nn
n = nn.LayerNorm(3)
input = torch.tensor([[10., 2., 12.],
                      [2., 3., 7.],
                      [100., 10., 1.],
                      [1., 1., 1.]], dtype=torch.float32)
print(input)
output = n(input)
print(output)

tensor([[ 10.,   2.,  12.],
        [  2.,   3.,   7.],
        [100.,  10.,   1.],
        [  1.,   1.,   1.]])
tensor([[ 0.4629, -1.3887,  0.9258],
        [-0.9258, -0.4629,  1.3887],
        [ 1.4094, -0.6040, -0.8054],
        [ 0.0000,  0.0000,  0.0000]], grad_fn=<NativeLayerNormBackward>)