local_response_normalization
local_response_normalization出现在论文”ImageNet Classification with deep Convolutional Neural Networks”中,论文中说,这种normalization对于泛化是有好处的.
经过了一个conv2d或pooling后,我们获得了[batch_size, height, width, channels]这样一个tensor.现在,将channels称之为层,不考虑batch_size
1 tf.nn.local_response_normalization(input, depth_radius=None, bias=None, alpha=None, beta=None, name=None)
2 '''
3 Local Response Normalization.
4 The 4-D input tensor is treated as a 3-D array of 1-D vectors (along the last dimension), and each vector is normalized independently. Within a given vector, each component is divided by the weighted, squared sum of inputs within depth_radius. In detail,
5 '''
6 """
7 input: A Tensor. Must be one of the following types: float32, half. 4-D.
8 depth_radius: An optional int. Defaults to 5. 0-D. Half-width of the 1-D normalization window.
9 bias: An optional float. Defaults to 1. An offset (usually positive to avoid dividing by 0).
10 alpha: An optional float. Defaults to 1. A scale factor, usually positive.
11 beta: An optional float. Defaults to 0.5. An exponent.
12 name: A name for the operation (optional).
13 """
举例子:
1 import tensorflow as tf
2
3 a = tf.constant([
4 [[1.0, 2.0, 3.0, 4.0],
5 [5.0, 6.0, 7.0, 8.0],
6 [8.0, 7.0, 6.0, 5.0],
7 [4.0, 3.0, 2.0, 1.0]],
8 [[4.0, 3.0, 2.0, 1.0],
9 [8.0, 7.0, 6.0, 5.0],
10 [1.0, 2.0, 3.0, 4.0],
11 [5.0, 6.0, 7.0, 8.0]]
12 ])
13 #reshape a,get the feature map [batch:1 height:2 width:2 channels:8]
14 a = tf.reshape(a, [1, 2, 2, 8])
15
16 normal_a=tf.nn.local_response_normalization(a,2,0,1,1)
17 with tf.Session() as sess:
18 print("feature map:")
19 image = sess.run(a)
20 print (image)
21 print("normalized feature map:")
22 normal = sess.run(normal_a)
23 print (normal)
feature map:
[[[[ 1. 2. 3. 4. 5. 6. 7. 8.]
[ 8. 7. 6. 5. 4. 3. 2. 1.]]
[[ 4. 3. 2. 1. 8. 7. 6. 5.]
[ 1. 2. 3. 4. 5. 6. 7. 8.]]]]
normalized feature map:
[[[[ 0.07142857 0.06666667 0.05454545 0.04444445 0.03703704 0.03157895
0.04022989 0.05369128]
[ 0.05369128 0.04022989 0.03157895 0.03703704 0.04444445 0.05454545
0.06666667 0.07142857]]
[[ 0.13793103 0.10000001 0.0212766 0.00787402 0.05194805 0.04
0.03448276 0.04545454]
[ 0.07142857 0.06666667 0.05454545 0.04444445 0.03703704 0.03157895
0.04022989 0.05369128]]]]
这里我取了n/2=2,k=0,α=1,β=1,举个例子,比如对于一通道的第一个像素“1”来说,我们把参数代人公式就是1/(1^2+2^2+3^2)=0.07142857,对于四通道的第一个像素“4”来说,公式就是4/(2^2+3^2+4^2+5^2+6^2)=0.04444445,以此类推
注意:这里的feature_map为【1,2,2,8】,其中1代表图像的数量,2X2代表图像的长宽,8代表图像的层数(map),NRL主要是利用map去计算,然后计算的值为图像的长宽(像素),与图像的数量无关!
我们可以这么理解,feature_map分割为直观的图像,第一个通道[1,8,4,1],第二个通道[2,7,3,2],第三个通道[3,6,2,3],以此类推。。。
那么求解的过程和上面就一一对应了,其中在边角达不到n的时候,那就省略。
能感觉到这种方法不好吗?效果肯定有的,因为对像素归一化了,有利于计算。但是对于一整幅图像来说反而没有什么太大的作用,因为归一化的种类不同,造成部分特征体现不出来,有时候反而不好。
batch_normalization
为什么有batch?上面结尾已经做了初步分析,也已经大概引出来对批量的图像做归一化,对单个图像做的归一化效果不好!
可以看出,batch_normalization之后,数据的维数没有任何变化,只是数值发生了变化 OutOut作为下一层的输入
函数:
tf.nn.batch_normalization()
def batch_normalization(x,
mean,
variance,
offset,
scale,
variance_epsilon,
name=None):
Args:
- x: Input
Tensor
of arbitrary dimensionality. - mean: A mean
Tensor
. - variance: A variance
Tensor
. - offset: An offset
Tensor
, often denoted ββ in equations, or None. If present, will be added to the normalized tensor. - scale: A scale
Tensor
, often denoted γγ in equations, orNone
. If present, the scale is applied to the normalized tensor. - variance_epsilon: A small float number to avoid dividing by 0.
- name: A name for this operation (optional).
- Returns: the normalized, scaled, offset tensor. 对于卷积,x:[bathc,height,width,depth] 对于卷积,我们要feature map中共享 γiγi 和 βiβi ,所以 γ,βγ,β的维度是[depth]
现在,我们需要一个函数 返回mean和variance, 看下面.
tf.nn.moments()
def moments(x, axes, shift=None, name=None, keep_dims=False):
# for simple batch normalization pass `axes=[0]` (batch only):
对于卷积的batch_normalization, x 为[batch_size, height, width, depth],axes=[0,1,2],就会输出(mean,variance), mean 与 variance 均为标量。
参考:
https://blog.csdn.net/mao_xiao_feng/article/details/53488271
https://www.jianshu.com/p/c06aea337d5d
https://blog.csdn.net/u012436149/article/details/52985303