Group Norm 学习笔记

论文：《Group Normalization》
地址：https://arxiv.org/pdf/1803.08494.pdf
优势：小批量训练时优势明显（相比另三个 Norm）
其他：Batch Norm、Layer Norm、Instance Norm

方法

将通道分组，在每组做 Norm

代码

源自论文

def GroupNorm(x, gamma, beta, G, eps=1e−5):
	# x: input features with shape [N,C,H,W]
	# gamma, beta: scale and offset, with shape [1,C,1,1] 
	# G: number of groups for GN
	N, C, H, W = x.shape
	x = tf.reshape(x, [N, G, C // G, H, W])
	
	mean, var = tf.nn.moments(x, [2, 3, 4], keep dims=True) 
	x = (x − mean) / tf.sqrt(var + eps)
	
	x = tf.reshape(x, [N, C, H, W]) 
	
	return x ∗ gamma + beta

实验