Batch Normalization 批标准化及其相关数学原理和推导

数据经过一层层网络之后，输出的数据分布会发生变化，此现象称为Internal Covariate Shift，会给下一层的网络学习带来困难：

1. 整个网络的学习速度较慢；
2. 数据的分布容易陷入激活函数（如sigmoid，tanh）的梯度饱和区，减慢了网络的收敛速度。

直接对每一层做归一化是不合理的

如果将每一层的输出都归一化为标准正态分布，均值为0，方差为1，会导致网络完全学习不到输入数据的特征，因为所有的特征都被归一化了。

底层网络学习到的参数信息被丢掉了，降低了网络的数据表达能力。

Batch Normalization的步骤 ¹

1. 求出整个batch数据的均值：
   $${\mu }_b=\frac{1}{m}\sum _{i=1}^m{x}_i$$

2. 求出整个batch数据的方差：
   $${\sigma }_b^2=\frac{1}{m}\sum _{i=1}^m(x_i-{\mu }_b{)}^2$$

3. 将数据归一化为标准正态分布：
   $${\hat{x}}_i=\frac{x_i-{\mu }_b}{\sqrt{{\sigma }_b^2+\epsilon }}$$
   其中的$\epsilon$是为了防止方差为0。

4. 引入平移和缩放参数，得到最终的归一化结果：
   $$y_i=\gamma {\hat{x}}_i+\beta$$

测试阶段使用Unbiased variance estimate ² 来进行总体方差的无偏估计

Question: Is the estimate of the population variance that arises in this way using the sample mean always smaller than what we would get if we used the population mean?
Answer: yes except when the sample mean happens to be the same as the population mean.

We are seeking the sum of squares of distances from the population mean, but end up calculating the sum of squares of differences from the sample mean, which, as will be seen, is the number that minimizes that sum of squares of distances. So unless the sample happens to have the same mean as the population, this estimate will always underestimate the sum of squared differences from the population mean.

证明过程如下：

设样本的平均值是${\mu }_s$，设总体的平均值是$\mu$，则
$${\mu }_s=\frac{1}{m}i=$$