tensorflow中Batch Normalization和Layer Normalization的实现

深度学习是一门经验主义的科学，也就是说，人们总是实验效果好了之后才尝试去解释。BN和LN进来得到这么广泛的应用，也不是因为其想法多么巧妙，而是因为其效果确实好。

tensorflow1.12版本中这两个函数的实现如下：

tf.contrib.layers.batch_norm(
    inputs,
    decay=0.999,
    center=True,
    scale=False,
    epsilon=0.001,
    activation_fn=None,
    param_initializers=None,
    param_regularizers=None,
    updates_collections=tf.GraphKeys.UPDATE_OPS,
    is_training=True,
    reuse=None,
    variables_collections=None,
    outputs_collections=None,
    trainable=True,
    batch_weights=None,
    fused=None,
    data_format=DATA_FORMAT_NHWC,
    zero_debias_moving_mean=False,
    scope=None,
    renorm=False,
    renorm_clipping=None,
    renorm_decay=0.99,
    adjustment=None
)

tf.contrib.layers.layer_norm(
    inputs,
    center=True,