关于在神经网络训练中使用批量归一化（batch_normalization）时遇到的参数

批量归一化

def batch_normalization(inputs,
                        axis=-1,
                        momentum=0.99,
                        epsilon=1e-3,
                        center=True,
                        scale=True,
                        beta_initializer=init_ops.zeros_initializer(),
                        gamma_initializer=init_ops.ones_initializer(),
                        moving_mean_initializer=init_ops.zeros_initializer(),
                        moving_variance_initializer=init_ops.ones_initializer(),
                        beta_regularizer=None,
                        gamma_regularizer=None,
                        beta_constraint=None,
                        gamma_constraint=None,
                        training=False,
                        trainable=True,
                        name=None,
                        reuse=None,
                        renorm=False,
                        renorm_clipping=None,
                        renorm_momentum=0.99,
                        fused=None,
                        virtual_batch_size=None,
                        adjustment=None):

参数 momentum

移动平均线的动量

momentum: Momentum for the moving average.

参数 epsilon

小的浮点数添加到方差中避免除以0

Small float added to variance to avoid dividing by zero.

参数 training

要么是Python布尔值，要么是TensorFlow布尔标量张量(如一个占位符)。是否在训练模式下返回输出(使用当前批次的统计数据进行规范化)或推理模式(使用移动统计数据进行标准化)。注:请务必设置此选项参数正确，否则您的训练/推理将不起作用正常。

training: Either a Python boolean, or a TensorFlow boolean scalar tensor
      (e.g. a placeholder). Whether to return the output in training mode
      (normalized with statistics of the current batch) or in inference mode
      (normalized with moving statistics). **NOTE**: make sure to set this
      parameter correctly, or else your training/inference will not work
      properly.

实例

training = tf.placeholder_with_default(False, shape=(), name='training')

hidden1 = tf.layers.dense(X, n_hidden1, name="hidden1") #全连接层，在DNN中视为隐藏层也可
bn1 = tf.layers.batch_normalization(hidden1, training=training, momentum=0.9) #批量归一化,tensorflow使用batch_normalization（）函数来实现中心化和归一化输入，但是你必须自己计算均值和标准方差
bn1_act = tf.nn.elu(bn1) #计算线性指数函数，对输入值进行处理

hidden2 = tf.layers.dense(bn1_act, n_hidden2, name="hidden2") #全连接层
bn2 = tf.layers.batch_normalization(hidden2, training=training, momentum=0.9)