Weakly- and Semi-Supervised Learning of a Deep Convolutional Network for Semantic Image Segmentation

abstract

Deep convolutional neural networks (DCNNs) trained on a large number of images with strong pixel-level annotations have recently significantly pushed the state-of-art in semantic image segmentation. We study the more challenging problem of learning DCNNs for semantic image segmentation from either

1. weakly annotated training data such as bounding boxes or image-level labels or
2. a combination of few strongly labeled and many weakly labeled images, sourced from one or multiple datasets.

We develop Expectation-Maximization (EM) methods for semantic image segmentation model training under these weakly supervised and semi-supervised settings. Extensive experimental evaluation shows that the proposed techniques can learn models delivering competitive results on the challenging PASCAL VOC 2012 image segmentation benchmark, while requiring significantly less annotation effort. We share source code implementing the proposed system at https://bitbucket.org/deeplab/deeplab-public.

Proposed Methods

像素级别的标注（全监督）

目标函数是：
$$J(\mathbf{\theta })=\log P(\mathbf{y}|\mathbf{x};\mathbf{\theta })=\sum _{m=1}^M\log P\left(y_m|\mathbf{x};\mathbf{\theta }\right)$$
式中，$\theta$是DNN的参数，每个像素的标签分布可以按照下式计算：
$$P\left(y_m|\mathbf{x};\mathbf{\theta }\right)\propto \exp \left(f_m\left(y_m|\mathbf{x};\mathbf{\theta }\right)\right)$$
式中，$f_m\left(y_m|\mathbf{x};\mathbf{\theta }\right)$是DCNN在第$m$个像素的输出，使SGD即可优化$J(\mathbf{\theta })$

图象级别的标注

当只有图像级注释可用时，我们可以观察到图像像素值$\mathbf{x}$和图像级标签$\mathbf{z}$，但像素级分割结果$\mathbf{y}$是潜在变量。建立如下概率图模型：
$$P(\mathbf{x},$$