gated convolution

1 Local response normalization（局部响应归一化）

1. 局部响应归一化，由于做了平滑处理，可以增加泛化能力，公式如下
   $$b_{x,y}^i=a_{x,y}^i/\left(bias+\alpha \sum _{j=\max (0,i-n/2)}^{\min (N-1,i+n/2)}{(a_{x,y}^i)}^2\right)$$
   其中i代表在第几个feature map，(x,y)代表所在feature map位置
2. tensorflow有内置函数tf.nn.lrn，参数输入与解释如下

"""Local Response Normalization.

  The 4-D `input` tensor is treated as a 3-D array of 1-D vectors (along the last
  dimension), and each vector is normalized independently.  Within a given vector,
  each component is divided by the weighted, squared sum of inputs within
  `depth_radius`.  In detail,

      sqr_sum[a, b, c, d] =
          sum(input[a, b, c, d - depth_radius : d + depth_radius + 1] ** 2)
      output = input / (bias + alpha * sqr_sum) ** beta
  a is batch size. d is channel.

2 gated convolution

公式如下
$$y(\mathbf{X})=(\mathbf{X}\ast \mathbf{W}+\mathbf{b})\otimes \sigma (\mathbf{X}\ast \mathbf{V}+\mathbf{c})$$
其中 W，V为两个不同的卷积核
tensorflow实现方法如下

def gate_conv(x_in, cnum, ksize, stride=1, rate=1, name='conv',
              padding='SAME', activation='leaky_relu', use_lrn=True, training=True):
    assert padding in ['SYMMETRIC', 'SAME', 'REFELECT']
    if padding == 'SYMMETRIC' or padding == 'REFELECT':
        p = int(rate * (ksize - 1) / 2)
        x = tf.pad(x_in, [[0, 0], [p, p], [p, p], [0, 0]], mode=padding)
        padding = 'VALID'
    x = tf.layers.conv2d(
        x_in, cnum, ksize, stride, dilation_rate=rate,
        activation=None, padding=padding, name=name)
    if use_lrn:
        x = tf.nn.lrn(x, bias=0.00005)
    if activation == 'leaky_relu':
        x = tf.nn.leaky_relu(x)

    g = tf.layers.conv2d(
        x_in, cnum, ksize, stride, dilation_rate=rate,
        activation=tf.nn.sigmoid, padding=padding, name=name + '_g')

    x = tf.multiply(x, g)
    return x, g