Batch Normalization导数计算与代码实现

最新推荐文章于 2024-07-30 12:10:19 发布

stesha_chen

最新推荐文章于 2024-07-30 12:10:19 发布

阅读量3k

点赞数 3

分类专栏：深度学习入门文章标签： batch normalization

本文链接：https://blog.csdn.net/stesha_chen/article/details/84999511

版权

本文详细介绍了Batch Normalization(BN)的导数计算过程，包括正向传播和反向传播的推导，并给出了TensorFlow中的源码实现。在正向计算中，强调了训练和测试阶段的区别，以及如何使用running_mean和running_var。反向计算部分，解释了如何根据cache中的信息和损失函数的梯度求解各参数的梯度。最后，展示了导数公式的简化及其在代码实现中的应用。

摘要由CSDN通过智能技术生成

关于batch normalization的论文阅读可以参考以前的一篇博文Batch Normalization，这里主要对BN的导数进行推导，并且看一下tensorflow中的源码实现。

BN导数推导

正向计算

回顾一下Batch Normalization的正向计算公式：

$\mu=\frac{1}{m}\sum_{i=1}^{m}{x_i}$ (1)

$\sigma^2=\frac{1}{m}\sum_{i=1}^{m}{(x_i - \mu)^2}$ (2)

$\hat{x_i} = \frac{x_i-\mu}{\sqrt{\sigma^2+\epsilon}}$ (3)

$\hat{y_i} = \gamma \hat{x_i}+\beta$ (4)

自己实现这个功能代码如下：

def batchnorm_forward(x, gamma, beta, bn_param):
    """
    Forward pass for batch normalization.

    During training the sample mean and (uncorrected) sample variance are
    computed from minibatch statistics and used to normalize the incoming data.
    During training we also keep an exponentially decaying running mean of the
    mean and variance of each feature, and these averages are used to normalize
    data at test-time.

    At each timestep we update the running averages for mean and variance using
    an exponential decay based on the momentum parameter:

    running_mean = momentum * running_mean + (1 - momentum) * sample_mean
    running_var = momentum * running_var + (1 - momentum) * sample_var

    Note that the batch normalization paper suggests a different test-time
    behavior: they compute sample mean and variance for each feature using a
    large number of training images rather than using a running average. For
    this implementation we have chosen to use running averages instead since
    they do not require an additional estimation step; the torch7
    implementation of batch normalization also uses running averages.

    Input:
    - x: Data of shape (N, D)
    - gamma: Scale parameter of shape (D,)
    - beta: Shift paremeter of shape (D,)
    - bn_param: Dictionary with the following keys:
      - mode: 'train' or 'test'; required
      - eps: Constant for numeric stability
      - momentum: Constant for running mean / variance.
      - running_mean: Array of shape (D,) giving running mean of features
      - running_var Array of shape (D,) giving running variance of features

    Returns a tuple of:
    - out: of shape (N, D)
    - cache: A tuple of values needed in the backward pass
    """
    mode = bn_param['mode']
    eps = bn_param.get('eps', 1e-5)
    momentum = bn_param.get('momentum', 0.9)

    N, D = x.shape
    running_mean = bn_param.get('running_mean', np.zeros(D, dtype=x.dtype))
    running_var = bn_param.get('running_var', np.zeros(D, dtype=x.dtype))

    out, cache = None, None
    if mode == 'train':
        #######################################################################
        # TODO: Implement the training-time forward pass for batch norm.      #
        # Use minibatch statistics to compute the mean and variance, use      #
        # these statistics to normalize the incoming data, and scale and      #
        # shift the normalized data using gamma and beta.                     #
        #                                                                     #
        # You should store the output in the variable out. Any intermediates  #
        # that you need for the backward pass should be stored in the cache   #
        # variable.                                                           #
        #                                                                     #
        # You should also use your computed sample mean and variance together #
        # with the momentum variable to update the running mean and running   #
        # variance, storing your result in the running_mean and running_var