【DropBlock】Deep Learning Technology

The paper records the newest deep learning technology

一、 DropBlock

1. Theory and Practice

• Choose Anchor to be dropped:
  • 操作：选中(feat_size - block_size -1)的中间区域，使用γ作为伯努利分布的均值，随机需要drop的点，用❌表示选中(feat_size - block_size -1)的中间区域，使用γ作为伯努利分布的均值，随机需要drop的点，用❌表示
  • 原理：Equation（1）
• Expand zero entry to zero block
  • 操作：❌周围[block_size block_size]的box也视为要drop的点❌周围[block_size block_size]的box也视为要drop的点
  • 原理：dropping continuous regions can remove certain semantic> dropping continuous regions can remove certain semantic information (e.g., head or feet) and consequently enforcing remaining units to learn features for
    classifying input image.
  • 效果：Figure 1
• Apply Mask and Normalize
  • 操作：step 7，8 in Algorithm 1 DropBlock

2. Paper and Implements

• paper
• keras, tf, pytorch,caffe
• 谷歌大脑提出DropBlock卷积正则化方法，显著改进CNN精度

3. Other Dropout Strategies

• Dropout：完全随机扔
• SpatialDropout：按channel随机扔
• Stochastic Depth：按res block随机扔
• DropBlock：每个feature map上按spatial块随机扔
• Cutout：在input层按spatial块随机扔
• DropConnect：只在连接处扔，神经元不扔。

4. Tensorflow

github上给出的dropblock-tensorflow版本（tf）亲测发现仅仅只能用于前馈（因为必须要求输入的维度已知，而不能为None）；keras版跟tf结合的时候也会碰到一些问题。
因此，这里基于网上的tf实现，给出可以在训练中实际使用的tensorflow版本的dropBlock，亲测对于psnr带来0.02到0.04不同程度的提升
代码如下：

import tensorflow as tf

def dropblock(x, keep_prob, block_size, training=None, scale=True):
    def drop():
        _, w, h, c = tf.shape(x)[0], tf.shape(x)[1], tf.shape(x)[2], tf.shape(x)[3]
        w, h = tf.to_float(w), tf.to_float(h)
        gamma = (1. - keep_prob) * (w * h) / (block_size ** 2) / ((w - block_size + 1) * (h - block_size + 1))
        w, h = tf.to_int32(w), tf.to_int32(h)
        sampling_mask_shape = tf.stack([1, h - block_size + 1, w - block_size + 1, c])
        noise_dist = tf.distributions.Bernoulli(probs=gamma)
        mask = noise_dist.sample(sampling_mask_shape)

        br = (block_size - 1) // 2
        tl = (block_size - 1) - br
        pad_shape = [[0, 0], [tl, br], [tl, br], [0, 0]]
        mask = tf.pad(mask, pad_shape)
        mask = tf.nn.max_pool(mask, [1, block_size, block_size, 1], [1, 1, 1, 1], 'SAME')
        mask = tf.cast(1 - mask, tf.float32)
        output = tf.multiply(x, mask)
        output = tf.cond(tf.constant(scale, dtype=tf.bool) if isinstance(scale, bool) else scale,
                     true_fn=lambda: output * tf.to_float(tf.size(mask)) / tf.reduce_sum(mask),
                     false_fn=lambda: output)
        return output

    output = tf.cond(tf.logical_or(tf.logical_not(training), tf.equal(keep_prob, 1.0)),
                     true_fn=lambda: x, false_fn=drop)
    return output

h = dropblock(x=h, keep_prob=self.dropout, block_size=3, training=self.is_training, scale=True)