论文:《Group Normalization》
地址:https://arxiv.org/pdf/1803.08494.pdf
优势:小批量训练时优势明显(相比另三个 Norm)
其他:Batch Norm、Layer Norm、Instance Norm
方法
将通道分组,在每组做 Norm
代码
源自论文
def GroupNorm(x, gamma, beta, G, eps=1e−5):
# x: input features with shape [N,C,H,W]
# gamma, beta: scale and offset, with shape [1,C,1,1]
# G: number of groups for GN
N, C, H, W = x.shape
x = tf.reshape(x, [N, G, C // G, H, W])
mean, var = tf.nn.moments(x, [2, 3, 4], keep dims=True)
x = (x − mean) / tf.sqrt(var + eps)
x = tf.reshape(x, [N, C, H, W])
return x ∗ gamma + beta
实验