BatchNormal推导和yolo源码解析

最新推荐文章于 2024-08-16 15:32:34 发布

蜜丝特潮

最新推荐文章于 2024-08-16 15:32:34 发布

阅读量3.7k

点赞数 15

分类专栏：深度学习 yolo源码分析

本文链接：https://blog.csdn.net/qq_29381089/article/details/80406428

版权

深度学习同时被 2 个专栏收录

5 篇文章 0 订阅

订阅专栏

yolo源码分析

4 篇文章 1 订阅

订阅专栏

最近一直在看yolo算法,感觉有时候只有看懂各种算法底层的实现,才能真正了解这个算法,这不,结合yolo源码和网上各种算法讲解,各种深度算法不再是模糊的印象了,而这里总结了最近看的BatchNormal

1.batchnormal解决了什么问题

1)梯度消失问题
对于网络任一层l,输入 $a^{l-1}$ ,一般都会经历一下过程:
$z=wa^{l-1}+b$
输出: $a^l=\sigma(z)$ ,
那么问题来了,假设激活函数 $\sigma$ 为逻辑函数:

如果某层的z分布在逻辑函数两边的接近水平的地方,也就是梯度接近0的位置,那么网络反向传播这层后,梯度降为接近0,梯度消失,那么参数,也就基本不更新了.
这里写图片描述

2)参数0均值初始化导致的问题
由于初始化的时候，参数一般都是0均值的，因此开始的拟合y=Wx+b，基本过原点附近，如图b红色虚线。因此，网络需要经过多次学习才能逐步达到如紫色实线的拟合，即收敛的比较慢。如果我们对输入数据先作减均值操作，如图c，显然可以加快学习。更进一步的，我们对数据再进行去相关操作，使得数据更加容易区分，这样又会加快训练，如图d。而batchnormal也有这种效果

3)内部迁移(Internal Covariate Shift)
数据经过多层神经网络后,数据分布发生变化,导致各层参数需要不到调整适应分布变化,这会让收敛速度变慢,超参数设定也变得比较复杂,这在论文里作者称作Internal Covariate Shift

那么batchnormal具体如何解决上述的呢,首先了解前行传播

`注意以下字母皆为矩阵形式,i为batch里面的索引`

2.BN前向传播

这里写图片描述
设有 $mini$ _ $batch$ 的batch_size为m,注意这里的i为batch里面的索引,x和a为特征矩阵
由于当前层的输入等于上一层的输出,那么设第l层输出为 $a^l$ ,则第l层的输入 $x^l_i=a^{l-1}_i,$ i =0,1,2…m
设

前向传播过程:
1.全连接则乘权重,卷积层则对x卷积

z l i = (w l i) T a l - 1 i / / 偏 重 放 后 面 加 (1.1)

$z^l_i=(w^l_i)^Ta^{l-1}_i\tag{1.1}\quad//偏重放后面加$
2.计算batch_size个z的均值

μ l B = 1 m \sum i = 1 m z l i (1.2)

$\mu^l_B=\frac 1 m \sum_{i=1}^{m}{z_i^l} \tag{1.2}$
3)计算batch_size个z的方差:

(D 2 B) (l) = 1 m \sum i = 1 m (z l i - μ l B) 2 (1.3)

$(D_B^2)^{(l)}=\frac1 m\sum_{i=1}^m{(z^l_i-\mu^l_B)^2}\tag{1.3}$
3)将batch_size个z,归一化成均值为0,方差为1的分布:

B N = z ̂ l i = z l i - μ l B ( D 2 B ) ( l ) + ε ‾ ‾ ‾ ‾ ‾ ‾ ‾ ‾ ‾ ‾ \sqrt (1.4)

$BN=\hat{z}^l_i=\frac {z^l_i-\mu^l_B}{\sqrt{(D_B^2)^{(l)}+\varepsilon}}\tag{1.4}$

经过这个操作后数据就被分布在0为圆心,1为半径的范围内了,这样以上问题就被成功解决了问题
这里写图片描述

4)放缩和迁移:

y l i = γ z ̂ l i + β (1.5)

$y^l_i=\gamma \hat{z}^l_i+\beta \tag{1.5}$
这步的作用在于,以逻辑函数为例,经过1.3后,数据主要分布在线性区域,非线性表达能力会受到影响,所以通过对数据放大或缩小和迁移来进入非线性区域范围
5)激活,输出

a l i = σ (y l i) (1.6)

$a^l_i=\sigma(y^l_i)\tag{1.6}$

yolo的前行传播源码：

void forward_batchnorm_layer(layer l, network net)
{
    //如果是batchnormal层,则直接输出等于输入
    if(l.type == BATCHNORM) copy_cpu(l.outputs*l.batch, net.input, 1, l.output, 1);
    //全链接层，看成通道数为l.outputs,特征图长宽为1
    if(l.type == CONNECTED){
        l.out_c = l.outputs;
        l.out_h = l.out_w = 1;
    }

    //l.x=l.output,如果按方差求导最终的化简等于0,来计算,l.x后面就用不到了
    copy_cpu(l.outputs*l.batch, l.output, 1, l.x, 1);

    //训练状态
    if(net.train){
        //求当前batch的均值，对应公式1.2
        mean_cpu(l.output, l.batch, l.out_c, l.out_h*l.out_w, l.mean);
        //求当前batch的方差，对应公式1.3
        variance_cpu(l.output, l.mean, l.batch, l.out_c, l.out_h*l.out_w, l.variance);
        //求均值的滚动平均，预测时,均值的就是这个值,什么是滚动平均，见下面注
        scal_cpu(l.out_c, .99, l.rolling_mean, 1);
        axpy_cpu(l.out_c, .01, l.mean, 1, l.rolling_mean, 1);

        //求方差的滚动平均,预测时,方差用的就是这个值,可以看非训练状态时normalize_cpu()函数的实现和参数
        scal_cpu(l.out_c, .99, l.rolling_variance, 1);
        axpy_cpu(l.out_c, .01, l.variance, 1, l.rolling_variance, 1);

        //对应公式1.4
        normalize_cpu(l.output, l.mean, l.variance, l.batch, l.out_c, l.out_h*l.out_w);   

        //将1.4式子的结果保存到l.x_norm,用于反向传播时相关参数梯度的计算
        copy_cpu(l.outputs*l.batch, l.output, 1, l.x_norm, 1);
    } 
    //非训练状态，如预测时
    else {
        //对应公式1.4
        normalize_cpu(l.output, l.rolling_mean, l.rolling_variance, l.batch, l.out_c, l.out_h*l.out_w);
    }
    //这两步,对应公式1.5,这里l.scale对应gamma,l.biases对应beta
    scale_bias(l.output, l.scales, l.batch, l.out_c, l.out_h*l.out_w);
    add_bias(l.output, l.biases, l.batch, l.out_c, l.out_h*l.out_w);
}

注：
滚动平均：

x ¯ n = x 1 + x 2 + \cdot \cdot \cdot + x m m = x 1 + x 2 + \cdot \cdot \cdot + x m - 1 + x m m = x 1 + x 2 + \cdot \cdot \cdot + x m - 1 m + x m m = ( x 1 + x 2 + \cdot \cdot \cdot + x m - 1 ) ( m - 1 ) m ( m - 1 ) + x m m = m - 1 m x ¯ m - 1 + x m m = (1 - 1 m) x ¯ m - 1 + 1 m x m

$\begin{align} \bar x_n&=\frac {x_1+x_2+\cdot\cdot\cdot+x_m}{m}\nonumber \\ &=\frac{x_1+x_2+\cdot\cdot\cdot+x_{m-1}+x_m}{m}\nonumber \\ &=\frac{x_1+x_2+\cdot\cdot\cdot+x_{m-1}}{m}+\frac{x_m}{m}\nonumber \\ &=\frac{(x_1+x_2+\cdot\cdot\cdot+x_{m-1})(m-1)}{m(m-1)}+\frac{x_m}{m}\nonumber \\ &=\frac{m-1}{m}\bar x_{m-1}+\frac {x_m}{m}\nonumber \\ &=(1-\frac 1 m)\bar x_{m-1}+\frac 1 mx_m \nonumber \\ \end{align}$

这里 $\bar x_n$ 表示前n个数据的平均值
yolo在这里直接将 $\frac 1 n=0.1$ ,简化计算,
至于为什么可以用这个均值来近似代替整个数据集的分分布,见后面的预测解释,推导

3.BN反向传播过程

首先推导几个值,为后面链式求导用:
设最终的损失函数为 $C$
对方差求导:

\begin{aligned} \frac{\partial C}{\partial (D_{B}^{2})^{l}} & = \sum_{i = 1}^{m} (\frac{\partial C}{\partial {\hat{z}}_{i}^{l}} \frac{\partial {\hat{z}}_{i}^{l}}{\partial (D_{B}^{2})^{l}}) \\ = \sum_{i = 1}^{m} {\frac{- 1}{2} \frac{\partial C}{\partial z_{i}^{l + 1}} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} \frac{\partial a_{i}^{l}}{\partial y_{i}^{l}} \frac{\partial y_{i}^{l}}{\partial {\hat{z}}_{i}^{l}} (z_{i}^{l} - μ_{B}^{l}) [(D_{B}^{2})^{(l)} + ε]^{- \frac{3}{2}}} \\ = \sum_{i = 1}^{m} {\frac{- 1}{2} [δ^{l + 1} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} ⨀ (σ (y_{i}^{l})^{'} \cdot γ_{i}^{l})] (z_{i}^{l} - μ_{B}^{l}) [(D_{B}^{2})^{(l)} + ε]^{- \frac{3}{2}}} \\ (2.1) & = γ^{l} ⨀ \sum_{i = 1}^{m} {\frac{- 1}{2} [δ^{l + 1} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} ⨀ σ (y_{i}^{l})^{'}] (z_{i}^{l} - μ_{B}^{l}) [(D_{B}^{2})^{(l)} + ε]^{- \frac{3}{2}}} \end{aligned}

$\begin{align} \frac {\partial C}{\partial (D^2_B)^l} &=\sum_{i=1}^{m}(\frac{\partial C}{\partial {\hat{z}^l_i}}\frac{\partial \hat{z}^l_i}{\partial (D^2_B)^l})\nonumber \\ &=\sum_{i=1}^{m}\lbrace\frac {-1} 2\frac{\partial C}{\partial {z}^{l+1}_i} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\frac{\partial a^l_i}{\partial y_i^l}\frac{\partial y_i^l}{\partial \hat{z}_i^l}{{(z^l_i-\mu^l_B)}}{[(D_B^2)^{(l)}+\varepsilon]^{-\frac 3 2}}\rbrace\nonumber \\ &=\sum_{i=1}^{m}\lbrace\frac {-1} 2[\delta^{l+1} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\bigodot(\sigma(y_i^l)'\cdot \gamma_i^l)]{{(z^l_i-\mu^l_B)}}{[(D_B^2)^{(l)}+\varepsilon]^{-\frac 3 2}}\rbrace\nonumber \\ &= \gamma^l\bigodot \sum_{i=1}^{m}\lbrace\frac {-1} 2[\delta^{l+1} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\bigodot \sigma(y_i^l)']{{(z^l_i-\mu^l_B)}}{[(D_B^2)^{(l)}+\varepsilon]^{-\frac 3 2}}\rbrace\nonumber \tag{2.1} \end{align}$

对均值求导:
这里需用到一个复合函数求导的方法:
这里写图片描述

所以按这个方法求导:

\partial C \partial μ l B = \sum i = 1 m (\partial C \partial z i ^z i ^ \partial μ l B + \partial C \partial D 2 B \partial D 2 B \partial μ l B) = \sum i = 1 m (\partial C \partial z i ^- 1 ( D 2 B ) l + ε ‾ ‾ ‾ ‾ ‾ ‾ ‾ ‾ ‾ \sqrt) + \partial C \partial ( D 2 B ) l \cdot - 2 m \cdot \sum i m (z l i - μ l B) = \sum i = 1 m (\partial C \partial z i ^- 1 ( D 2 B ) l + ε ‾ ‾ ‾ ‾ ‾ ‾ ‾ ‾ ‾ \sqrt) = \sum i = 1 m (\partial C \partial z l + 1 i \partial z l + 1 i \partial a l i \partial a l i \partial y l i \partial y l i \partial z ̂ l i - 1 ( D 2 B ) l + ε ‾ ‾ ‾ ‾ ‾ ‾ ‾ ‾ ‾ \sqrt) = γ l ⨀ \sum i = 1 m ([δ l + 1 \partial z l + 1 i \partial a l i ⨀ σ (y l i)'] - 1 ( D 2 B ) l + ε ‾ ‾ ‾ ‾ ‾ ‾ ‾ ‾ ‾ \sqrt) (2.2)

$\begin{align} {\frac{\partial C}{\partial \mu_B^l}}&=\sum_{i=1}^m(\frac {\partial C}{\partial \hat{z_i} }\frac { \hat{z_i}}{\partial \mu_B^l}+\frac{\partial C}{\partial D^2_B}\frac{\partial D^2_B}{\partial \mu_B^l})\nonumber \\ &=\sum_{i=1}^{m}(\frac {\partial C}{\partial \hat{z_i} }\frac{-1}{\sqrt{(D^2_B)^l+\varepsilon}})+\frac{\partial C}{\partial (D^2_B)^l}\cdot \frac {-2} m \cdot \sum_i^m{(z^l_i-\mu^l_B)}\nonumber \\ &=\sum_{i=1}^{m}(\frac {\partial C}{\partial \hat{z_i} }\frac{-1}{\sqrt{(D^2_B)^l+\varepsilon}})\nonumber \\ &=\sum_{i=1}^{m}(\frac{\partial C}{\partial {z}^{l+1}_i} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\frac{\partial a^l_i}{\partial y_i^l}\frac{\partial y_i^l}{\partial \hat{z}_i^l}\frac{-1}{\sqrt{(D^2_B)^l+\varepsilon}})\nonumber \\ &=\gamma^l \bigodot \sum_{i=1}^{m}([\delta^{l+1} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\bigodot \sigma(y_i^l)']\frac{-1}{\sqrt{(D^2_B)^l+\varepsilon}})\tag{2.2} \end{align}$

δ l i = \partial C \partial z l i = \partial C \partial z ̂ l i \partial z ̂ l i \partial z l i + \partial C \partial D 2 B \partial D 2 B \partial z l i + \partial C \partial μ B \partial μ B \partial z l i = \partial C \partial z ̂ l i 1 D 2 B + ε ‾ ‾ ‾ ‾ ‾ ‾ ‾ \sqrt + \partial C \partial D 2 B \cdot 2 m \cdot (z l i - μ B) + \partial C \partial μ B \cdot 1 m = \partial C \partial z l + 1 i \partial z l + 1 i \partial a l i \partial a l i \partial y l i \partial y l i \partial z ̂ l i 1 D 2 B + ε ‾ ‾ ‾ ‾ ‾ ‾ ‾ \sqrt + \partial C \partial D 2 B \cdot 2 m \cdot (z l i - μ B) + \partial C \partial μ B \cdot 1 m = [δ l + 1 \partial z l + 1 i \partial a l i ⨀ (σ (y l i)' ⨀ γ l i)] 1 D 2 B + ε ‾ ‾ ‾ ‾ ‾ ‾ ‾ \sqrt + \partial C \partial D 2 B \cdot 2 m \cdot (z l i - μ B) + \partial C \partial μ B \cdot 1 m (2.3)

$\begin{align} \delta^l_i=\frac{\partial C}{\partial z^l_i} &=\frac{\partial C}{\partial \hat{z}^l_i}\frac{\partial \hat{z}^l_i}{\partial z^l_i}+\frac{\partial C}{\partial D^2_B}\frac{\partial D^2_B}{\partial z^l_i}+\frac{\partial C}{\partial \mu_B}\frac {\partial \mu_B}{\partial z^l_i}\nonumber\\ &=\frac{\partial C}{\partial \hat{z}^l_i}\frac{1}{\sqrt{D^2_B+\varepsilon}}+\frac{\partial C}{\partial D^2_B}\cdot \frac 2 {m} \cdot (z^l_i-\mu_B)+\frac{\partial C}{\partial \mu_B}\cdot \frac 1 m\nonumber\\ &=\frac{\partial C}{\partial {z}^{l+1}_i} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\frac{\partial a^l_i}{\partial y_i^l}\frac{\partial y_i^l}{\partial \hat{z}_i^l}\frac{1}{\sqrt{D^2_B+\varepsilon}}+\frac{\partial C}{\partial D^2_B}\cdot \frac 2 {m} \cdot (z^l_i-\mu_B)+\frac{\partial C}{\partial \mu_B}\cdot \frac 1 m\nonumber\\ &=[\delta^{l+1} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\bigodot(\sigma(y_i^l)'\bigodot \gamma_i^l)]\frac{1}{\sqrt{D^2_B+\varepsilon}}+\frac{\partial C}{\partial D^2_B}\cdot \frac 2 {m} \cdot (z^l_i-\mu_B)+\frac{\partial C}{\partial \mu_B}\cdot \frac 1 m \tag{2.3} \end{align}$

则，以全连接层为例，求权重和偏差梯度：

\partial C \partial w l = \sum i = 1 m \partial C \partial z l i \partial z l i \partial w l = \sum i = 1 m (a l - 1) T δ l (2.4)

$\frac {\partial C}{\partial w^l}=\sum_{i=1}^{m}\frac{\partial C}{\partial z_i^l}\frac{\partial z_i^l}{\partial w^l}=\sum_{i=1}^m(a^{l-1})^T\delta^l\tag{2.4}$

\partial C \partial β l = \sum i = 1 m \partial C \partial z l + 1 i \partial z l + 1 i \partial a l i \partial a l i \partial y l i \partial y l i \partial β l = \sum i = 1 m δ l + 1 i \partial z l + 1 i \partial a l i ⨀ [σ (y l i)') \partial y l \partial β l] = \sum i = 1 m δ l + 1 i \partial z l + 1 i \partial a l i ⨀ σ (y l i)' (2.5)

$\frac{\partial C}{\partial \beta^l}=\sum_{i=1}^{m}\frac{\partial C}{\partial {z}_i^{l+1}}\frac{\partial {z}_i^{l+1}}{\partial a_i^l}\frac{\partial a_i^l}{\partial {y}_i^l}\frac {\partial {y}_i^l}{\partial \beta^l}=\sum_{i=1}^{m}\delta_i^{l+1} \frac{\partial {z}_i^{l+1}}{\partial a_i^l}\bigodot [\sigma(y_i^l)')\frac {\partial {y}^l}{\partial \beta^l}]=\sum_{i=1}^{m}\delta_i^{l+1} \frac{\partial {z}_i^{l+1}}{\partial a_i^l}\bigodot \sigma(y_i^l)'\tag{2.5}$

同理:

\partial C \partial γ l = \sum i = 1 m \partial C \partial z l + 1 i \partial z l + 1 i \partial a l i \partial a l i \partial y l i \partial y l i \partial γ l = \sum i = 1 m δ l + 1 i \partial z l + 1 i \partial a l i ⨀ [σ (y l i)' ⨀ z ̂ l i] (2.6)

$\frac {\partial C}{\partial \gamma^l}=\sum_{i=1}^{m}\frac{\partial C}{\partial {z}_i^{l+1}}\frac{\partial {z}_i^{l+1}}{\partial a_i^l}\frac{\partial a_i^l}{\partial {y}_i^l}\frac {\partial {y}_i^l}{\partial \gamma^l}=\sum_{i=1}^{m}\delta_i^{l+1} \frac{\partial {z}_i^{l+1}}{\partial a_i^l}\bigodot[ \sigma(y_i^l)'\bigodot \hat{z}_i^l]\tag{2.6}$

对应yolo代码:

void backward_batchnorm_layer(layer l, network net)
{
    //非训练状态
    if(!net.train){
        l.mean = l.rolling_mean;
        l.variance = l.rolling_variance;
    }
    //求偏差beta的梯度,对应公式2.5
    backward_bias(l.bias_updates, l.delta, l.batch, l.out_c, l.out_w*l.out_h);

    //求gamma梯度,对应公式2.6
    backward_scale_cpu(l.x_norm, l.delta, l.batch, l.out_c, l.out_w*l.out_h, l.scale_updates);


     //先计算公式2.3里面的公共项gamma
    scale_bias(l.delta, l.scales, l.batch, l.out_c, l.out_h*l.out_w);

    //求y对均值的导数,对应公式2.2
    mean_delta_cpu(l.delta, l.variance, l.batch, l.out_c, l.out_w*l.out_h, l.mean_delta);

    //求y对方差的导数,对应公式2.1,这里按上面化简后的公式,若激活函数为relu应该直接等于0
    variance_delta_cpu(l.x, l.delta, l.mean, l.variance, l.batch, l.out_c, l.out_w*l.out_h, l.variance_delta);

    //求权重的误差度,对应公式2.3
    normalize_delta_cpu(l.x, l.mean, l.variance, l.mean_delta, l.variance_delta, l.batch, l.out_c, l.out_w*l.out_h, l.delta);

     //对于BATCHNORM层,直接输出等于输入
    if(l.type == BATCHNORM) copy_cpu(l.outputs*l.batch, l.delta, 1, net.delta, 1);
}

4.预测：

预测时，计算总体的均值和方差是不实际的，也是无法实现的，因为无法采样到所有样本。用总采样来估计总体的均值和方差呢？也是需要大量计算的，在训练过程中的batch下的均值uB和方差σB，可以加以利用来估计总体
具体推导如下:

$E(X_i).D(X_i)$ :可以理解为 $X_i$ 所在分布的期望值

样本均值:

μ A = X ¯ = 1 m \sum i = 1 m X i (3.1)

$\mu_A=\bar X=\frac 1 m \sum_{i=1}^mX_i \tag{3.1}$
因为抽样和样本同分布,所以:
样本期望:

μ = E (X i) = E (μ A) = E (x) (3.2)

$\mu=E(X_i)=E(\mu_A)=E(x)\tag{3.2}$
但

μ A = E (X A i) \neq E (X i) (3.3)

$\mu_A=E(X_{A_i})\neq E(X_i)\tag{3.3}$
样本方差:

σ2=D(Xi)=D(x)(3.4) (3.4) σ 2 = D ( X i ) = D ( x ) $\sigma^2=D(X_i)=D(x)\tag{3.4}$

所以: 所以 : $所以:$

μ=1n∑ni=1E(Xi)(3.5) (3.5) μ = 1 n ∑ i = 1 n E ( X i ) $\mu=\frac 1 n\sum_{i=1}^nE(X_i)\tag{3.5}$ $

均值和方差:

E (μ A) = E (1 m \sum i = 1 m X i) = 1 z m n \sum i = 1 m n X i = E (x) (3.6)

$E(\mu_A)=E(\frac 1 m \sum_{i=1}^mX_i)=\frac 1 zmn \sum_{i=1}^{mn}X_i=E(x)\tag{3.6}$

D (μ A) = D (1 m \sum i = 1 m X i) = 1 m 2 \sum i = 1 m D (X i) = 1 m D (x) (3.7)

$D(\mu_A)=D(\frac 1 m\sum_{i=1}^mX_i)=\frac 1 {m^2}\sum_{i=1}^mD(X_i)=\frac 1 m D(x)\tag{3.7}$

E (σ 2 A) = E [\sum i = 1 m (X i - μ A) 2 \cdot 1 m] = 1 m \cdot E {\sum i = 1 m [X i - E (x) + E (x) - μ A] 2} = 1 m \cdot E {\sum i = 1 m [(X i - E (x)) 2 - 2 (X i - E (x) (μ A - E (x) + (μ A - E (x)) 2]} = 1 m \cdot E {\sum i = 1 m [(X i - E (x)) 2] - 2 m (μ A - E (x)) 2 + m (μ A - E (x)) 2} = 1 m \cdot E {\sum i = 1 m [(X i - E (x)) 2] - m (μ A - E (x)) 2} = D (x) - E {[μ A - E (μ A)] 2} = D (x) - D (μ A) = m - 1 m D (x) (1)

$\begin{align} E(\sigma^2_A) &=E[\sum_{i=1}^m(X_i-\mu_A)^2\cdot \frac 1 m] \nonumber \\ &=\frac 1 m \cdot E\lbrace\sum_{i=1}^m[X_i-E(x)+E(x)-\mu_A]^2\rbrace\nonumber \\ &=\frac 1 m \cdot E\lbrace\sum_{i=1}^m[(X_i-E(x))^2-2(X_i-E(x)(\mu_A-E(x)+(\mu_A-E(x))^2]\rbrace\nonumber \\ &=\frac 1 m \cdot E\lbrace\sum_{i=1}^m[(X_i-E(x))^2]-2m(\mu_A-E(x))^2+m(\mu_A-E(x))^2\rbrace\nonumber \\ &=\frac 1 m \cdot E\lbrace\sum_{i=1}^m[(X_i-E(x))^2]-m(\mu_A-E(x))^2\rbrace\nonumber \\ &=\ D(x)- E\lbrace[\mu_A-E(\mu_A)]^2\rbrace\nonumber \\ &=D(x)-D(\mu_A)\nonumber \\ &=\frac {m-1} mD(x)\end{align}\nonumber \\$

即: $D(x)=\frac m {m-1}E(\sigma^2_A)\tag{3.8}$

所以最终结果:
$\mu =E(x)=E(\mu_B)=\frac 1 K \sum_B^K \mu_B\tag{K为总迭代次数}$
$D(x)=\frac m {m-1}E(\sigma^2_A)\tag{3.8}$ tag{m为batch_size}$

虽然理论上是这样的,但yolo里面好像用滚动平均值来算,而不是上面的计算方法这部分计算对应前行传播函数forward_batchnorm_layer()里的:

scal_cpu(l.out_c, .99, l.rolling_mean, 1);
axpy_cpu(l.out_c, .01, l.mean, 1, l.rolling_mean, 1);
scal_cpu(l.out_c, .99, l.rolling_variance, 1);
axpy_cpu(l.out_c, .01, l.variance, 1, l.rolling_variance, 1);

Bacthnormal优化办法

1)增大学习率.
BN能减少每层的梯度变化幅度,使梯度稳定在理想的变化范围内,所以大学习率一般不会导致梯度消失
另外大学习率,训练一次,一般会导致参数,如权重变大,假设变大了n倍,即 $W'=nW$ ,其中W’为目前权重,W’为上次训练的权重

设上一次训练,均值为:
$\mu_B=\frac 1 m\sum_{i=1}^mWa^{l-1}$
方差:
$D_B^2=\frac 1 m\sum_{i=1}^m(Wa^{l-1}-\mu_B)^2$

则本次均值:
$\mu_B'=\frac 1 m\sum_{i=1}^mnWa^{l-1}=n\mu_B$
$(D_B^2)'=\frac 1 m\sum_{i=1}^m(nWa^{l-1}-\mu_B')^2=n^2D_B^2$

所以:
$BN(Wa^{l-1})=BN(nWa^{l-1})$
$\frac{\partial BN(nWa^{l-1})}{\partial (nW)}=\frac 1 n\frac{\partial BN(Wa^{l-1})}{\partial W}$
即:
$\frac{\partial BN(W'a^{l-1})}{\partial (W')}=\frac 1 n\frac{\partial BN(Wa^{l-1})}{\partial W}$
可以看出,当因上一次训练的大学习率导致权重W变大n倍后,只会让本次训练的梯度更小,这样一点程度上,便避免了过大学习率导致的梯度爆炸.