tensorflow使用BN—Batch Normalization

你要的答案或许都在这里：小鹏的博客目录

MachineLP的Github（欢迎follow）：https://github.com/MachineLP

我的GitHub：https://github.com/MachineLP/train_cnn-rnn-attention 自己搭建的一个框架，包含模型有：vgg(vgg16,vgg19), resnet(resnet_v2_50,resnet_v2_101,resnet_v2_152), inception_v4, inception_resnet_v2等。

注意：不要随便加BN，有些问题加了后会导致loss变大。

上一篇是 Batch Normalization的原理介绍，看一下tf的实现，加到卷积后面和全连接层后面都可：

（1）

训练的时候：is_training为True。

import tensorflow as tf
import numpy as np
from tensorflow.python.ops import control_flow_ops
from tensorflow.python.training import moving_averages

def bn(x, is_training):
    x_shape = x.get_shape()
    params_shape = x_shape[-1:]

    axis = list(range(len(x_shape) - 1))

    beta = _get_variable('beta', params_shape, initializer=tf.zeros_initializer())
    gamma = _get_variable('gamma', params_shape, initializer=tf.ones_initializer())

    moving_mean = _get_variable('moving_mean', params_shape, initializer=tf.zeros_initializer(), trainable=False)
    moving_variance = _get_variable('moving_variance', params_shape, initializer=tf.ones_initializer(), trainable=False)

    # These ops will only be preformed when training.
    mean, variance = tf.nn.moments(x, axis)
    update_moving_mean = moving_averages.assign_moving_average(moving_mean, mean, BN_DECAY)
    update_moving_variance = moving_averages.assign_moving_average(moving_variance, variance, BN_DECAY)
    tf.add_to_collection(UPDATE_OPS_COLLECTION, update_moving_mean)
    tf.add_to_collection(UPDATE_OPS_COLLECTION, update_moving_variance)

    mean, variance = control_flow_ops.cond(
        is_training, lambda: (mean, variance),
        lambda: (moving_mean, moving_variance))

    return tf.nn.batch_normalization(x, mean, variance, beta, gamma, BN_EPSILON)

函数: 
tf.nn.batch_normalization()

def batch_normalization(x,
                        mean,
                        variance,
                        offset,
                        scale,
                        variance_epsilon,
                        name=None):

• 1
• 2
• 3
• 4
• 5
• 6
• 7

• 1
• 2
• 3
• 4
• 5
• 6
• 7

Args:

• x: Input Tensor of arbitrary dimensionality.
• mean: A mean Tensor.
• variance: A variance Tensor.
• offset: An offset Tensor, often denoted β in equations, or None. If present, will be added to the normalized tensor.
• scale: A scale Tensor, often denoted γ in equations, or None. If present, the scale is applied to the normalized tensor.
• variance_epsilon: A small float number to avoid dividing by 0.
• name: A name for this operation (optional).
• Returns: the normalized, scaled, offset tensor. 
  对于卷积,x:[bathc,height,width,depth] 
  对于卷积,我们要feature map中共享 γi 和 βi ,所以 γ,β的维度是[depth]

另外，这里有使用batch normalization的示例：martin-gorner/tensorflow-mnist-tutorial

还可以参考：resnet：https://github.com/MachineLP/tensorflow-resnet

还可以看大师之作：CNN和RNN中如何引入BatchNorm

训练好的模型加载：tensorflow中batch normalization的用法

如果帮到你了，请赞赏支持：