Semi-supervised Semantic Segmentation Papers

Semi-Supervised Object Detection with Adaptive Class-Rebalancing Self-Training
1.SSOD中的伪标签不准确。此外，在前景-背景不平衡的标记数据上训练的模型容易产生有偏差的预测。
2. 类别不平衡（前景背景不平衡，前景类别不平衡）
无代码
Semi-Supervised Object Detection with Adaptive Class-Rebalancing Self-Training
(2022 CVPR) U2PL：Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels
半监督语义分割的关键是为未标记图像的像素分配足够的伪标签。
即使是不可靠的预测结果，虽然无法打上确定的伪标签，但仍可以作为部分类别的负样本，从而参与到模型的训练，从而让所有的无标签样本都能在训练过程中发挥作用。

(2022 CVPR) Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation
prediction accuracy problem of consistency learning methods
不准确的预测将会导致：
1.基于“严格”交叉熵(CE)损失的训练很容易过拟合预测错误，导致确认偏差;
2.应用于这些不准确预测的扰动将使用潜在错误的预测作为训练信号，有损consistency learning。
对于无监督部分loss进行confidence加权，监督是两个教师模型的预测平均
一种新型的特征扰动称为T-VAT，它基于从MT模型的老师那里学习到的对抗性噪声，并应用于学生模型
(2022 CVPR) Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation
Bootstrapping Semantic Segmentation with Regional Contrast
ReCo 会在所有类别里 sample 容易混淆的类来辅助 segmentation 网络学习更好的 representation。
Bootstrapping Semantic Segmentation with Regional Contrast
(2022 CVPR) Semi-supervised Semantic Segmentation with Error Localization Network
https://link.zhihu.com/?target=https%3A//arxiv.org/abs/2204.02078
https://link.zhihu.com/?target=https%3A//github.com/kinux98/SSL_ELN
https://blog.csdn.net/zn961018/article/details/124267289
(2022 CVPR) Semi-supervised Semantic Segmentation with Error Localization Network
This paper studies semi-supervised learning of semantic segmentation, which assumes that only a small portion of training images are labeled and the others remain unlabeled. The unlabeled images are usually assigned pseudo labels to be used in training, which however often causes the risk of performance degradation due to the confirmation bias towards errors on the pseudo labels. We present a novel method that resolves this chronic issue of pseudo labeling. At the heart of our method lies error localization network (ELN), an auxiliary module that takes an image and its segmentation prediction as input and identifies pixels whose pseudo labels are likely to be wrong. ELN enables semi-supervised learning to be robust against inaccurate pseudo labels by disregarding label noises during training and can be naturally integrated with self-training and contrastive learning. Moreover, we introduce a new learning strategy for ELN that simulates plausible and diverse segmentation errors during training of ELN to enhance its generalization. Our method is evaluated on PASCAL VOC 2012 and Cityscapes, where it outperforms all existing methods in every evaluation setting.
本文研究了语义分割的半监督学习，该学习假设只有一小部分训练图像被标记，而其他图像保持未标记。未标记的图像通常被分配用于训练的伪标签，但由于对伪标签错误的确认偏差，这通常会导致性能下降的风险。 我们提出了一种新的方法来解决这个长期存在的伪标记问题。我们方法的核心是误差定位网络（ELN），它是一个辅助模块，将图像及其分割预测作为输入，并识别伪标签可能错误的像素。ELN通过在训练过程中忽略标签噪声，使半监督学习对不准确的伪标签具有鲁棒性，并且可以自然地与自训练和对比学习相结合。 此外，我们还引入了一种新的ELN学习策略，该策略模拟了ELN训练过程中可能出现的各种分割错误，以增强其泛化能力。我们的方法在PASCAL VOC 2012和Cityscapes上进行了评估，在每个评估环境中，它都优于所有现有方法。
(2021 CVPR) CPS: Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision
https://link.zhihu.com/?target=https%3A//arxiv.org/abs/2106.01226
https://github.com/charlesCXK/TorchSemiSeg
(2021 CVPR) CPS:Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision
In this paper, we study the semi-supervised semantic segmentation problem via exploring both labeled data and extra unlabeled data. We propose a novel consistency regularization approach, called cross pseudo supervision (CPS). Our approach imposes the consistency on two segmentation networks perturbed with different initialization for the same input image. The pseudo one-hot label map, output from one perturbed segmentation network, is used to supervise the other segmentation network with the standard cross-entropy loss, and vice versa. The CPS consistency has two roles: encourage high similarity between the predictions of two perturbed networks for the same input image, and expand training data by using the unlabeled data with pseudo labels. Experiment results show that our approach achieves the state-of-the-art semi-supervised segmentation performance on Cityscapes and PASCAL VOC 2012
本文通过对已标注数据和额外未标注数据的探索，研究了半监督语义分割问题。我们提出了一种新的一致性正则化方法，称为交叉伪监督(CPS)。我们的方法将一致性强加于两个被同一输入图像不同初始化扰动的分割网络上。从一个扰动的分割网络输出的伪一热标签映射被用来监督具有标准交叉熵损失的另一个分割网络，反之亦然。CPS一致性有两个作用：鼓励两个扰动网络对同一输入图像的预测之间具有较高的相似性，以及通过使用带有伪标签的未标记数据来扩展训练数据。

(2021 ICCV) Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank
This work presents a novel approach for semi-supervised semantic segmentation. The key element of this approach is our contrastive learning module that enforces the segmentation network to yield similar pixel-level feature representations for same-class samples across the whole dataset. To achieve this, we maintain a memory bank continuously updated with relevant and high-quality feature vectors from labeled data. In an end-to-end training, the features from both labeled and unlabeled data are optimized to be similar to same-class samples from the memory bank. Our approach outperforms the current state-of-the-art for semi-supervised semantic segmentation and semi-supervised domain adaptation on well-known public benchmarks, with larger improvements on the most challenging scenarios, i.e., less available labeled data. https://github.com/Shathe/SemiSeg-Contrastive
这项工作提出了一种新的半监督语义分割方法。 这种方法的关键元素是我们的对比学习模块，它强制分割网络为整个数据集的同类样本产生相似的像素级特征表示。 为了实现这一点，我们维护了一个记忆库，使用来自标记数据的相关和高质量的特征向量不断更新。 在端到端训练中，标记和未标记数据的特征都经过优化，以类似于记忆库中的同类样本。 我们的方法在众所周知的公共基准上优于当前最先进的半监督语义分割和半监督域适应，在最具挑战性的场景（即可用的标记数据较少）上有更大的改进。
(2021 Nips) Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning
https://arxiv.org/abs/2110.05474
https://github.com/hzhupku/SemiSeg-AEL
Due to the limited and even imbalanced data, semi-supervised semantic segmentation tends to have poor performance on some certain categories, e.g., tailed categories in Cityscapes dataset which exhibits a long-tailed label distribution. Existing approaches almost all neglect this problem, and treat categories equally. Some popular approaches such as consistency regularization or pseudo-labeling may even harm the learning of under-performing categories, that the predictions or pseudo labels of these categories could be too inaccurate to guide the learning on the unlabeled data. In this paper, we look into this problem, and propose a novel framework for semi-supervised semantic segmentation, named adaptive equalization learning (AEL). AEL adaptively balances the training of well and badly performed categories, with a confidence bank to dynamically track category-wise performance during training. The confidence bank is leveraged as an indicator to tilt training towards under-performing categories, instantiated in three strategies: 1) adaptive Copy-Paste and CutMix data augmentation approaches which give more chance for under-performing categories to be copied or cut; 2) an adaptive data sampling approach to encourage pixels from under-performing category to be sampled; 3) a simple yet effective re-weighting method to alleviate the training noise raised by pseudo-labeling. Experimentally, AEL outperforms the state-of-the-art methods by a large margin on the Cityscapes and Pascal VOC benchmarks under various data partition protocols.
由于数据有限甚至不平衡，半监督语义分割往往在某些特定类别上表现不佳，例如 Cityscapes 数据集中的有尾类别表现出长尾标签分布。现有的方法几乎都忽略了这个问题，对类别一视同仁。一些流行的方法，例如一致性正则化或伪标签，甚至可能会损害对表现不佳的类别的学习，因为这些类别的预测或伪标签可能太不准确，无法指导对未标记数据的学习。在本文中，我们研究了这个问题，并提出了一种新的半监督语义分割框架，称为自适应均衡学习（AEL）。 AEL 自适应地平衡了表现良好和表现不佳的类别的训练，并使用置信库在训练期间动态跟踪类别表现。置信度库被用作指标，将训练向表现不佳的类别倾斜，以三种策略实例化：1）自适应复制粘贴和 CutMix 数据增强方法，这为复制或剪切表现不佳的类别提供了更多机会； 2) 一种自适应数据采样方法，以鼓励对来自表现不佳类别的像素进行采样； 3）一种简单而有效的重新加权方法，以减轻伪标签引起的训练噪声。在实验上，AEL 在各种数据分区协议下的 Cityscapes 和 Pascal VOC 基准测试中大大优于最先进的方法
(2021 ICCV) Re-Distributing Biased Pseudo Labels for Semi-Supervised Semantic Segmentation: A Baseline Investigation
http://arxiv.org/abs/2107.11279
https://github.com/CVMI-Lab/DARS
While self-training has advanced semi-supervised semantic segmentation, it severely suffers from the longtailed class distribution on real-world semantic segmentation datasets that make the pseudo-labeled data bias toward majority classes. In this paper, we present a simple and yet effective Distribution Alignment and Random Sampling (DARS) method to produce unbiased pseudo labels that match the true class distribution estimated from the labeled data. Besides, we also contribute a progressive data augmentation and labeling strategy to facilitate model training with pseudo-labeled data. Experiments on both Cityscapes and PASCAL VOC 2012 datasets demonstrate the effectiveness of our approach. Albeit simple, our method performs favorably in comparison with state-of-the-art approaches.
虽然自我训练具有先进的半监督语义分割，但它严重受到现实世界语义分割数据集上的长尾类分布的影响，这使得伪标记数据偏向于多数类。 在本文中，我们提出了一种简单而有效的分布对齐和随机采样 (DARS) 方法来生成与从标记数据估计的真实类分布相匹配的无偏伪标签。 此外，我们还提供了一种渐进式数据增强和标记策略，以促进使用伪标记数据进行模型训练。 Cityscapes 和 PASCAL VOC 2012 数据集的实验证明了我们方法的有效性。 尽管很简单，但与最先进的方法相比，我们的方法表现得更好。
(2021 CVPR) Semi-supervised Semantic Segmentation with Directional Context-aware Consistency.
https://jiaya.me/papers/semiseg_cvpr21.pdf
https://github.com/Jia-Research-Lab/Context-Aware-Consistency
Semantic segmentation has made tremendous progress in recent years. However, satisfying performance highly depends on a large number of pixel-level annotations. Therefore, in this paper, we focus on the semi-supervised segmentation problem where only a small set of labeled data is provided with a much larger collection of totally unlabeled images. Nevertheless, due to the limited annotations, models may overly rely on the contexts available in the training data, which causes poor generalization to the scenes unseen before. A preferred high-level representation should capture the contextual information while not losing selfawareness. Therefore, we propose to maintain the contextaware consistency between features of the same identity but with different contexts, making the representations robust to the varying environments. Moreover, we present the Directional Contrastive Loss (DC Loss) to accomplish the consistency in a pixel-to-pixel manner, only requiring the feature with lower quality to be aligned towards its counterpart. In addition, to avoid the false-negative samples and filter the uncertain positive samples, we put forward two sampling strategies. Extensive experiments show that our simple yet effective method surpasses current state-of-the-art methods by a large margin and also generalizes well with extra image-level annotations.
近年来，语义分割取得了巨大的进步。然而，令人满意的性能高度依赖于大量的像素级注释。因此，在本文中，我们专注于半监督分割问题，其中只有一小部分标记数据提供了更大的完全未标记图像集合。然而，由于注释有限，模型可能过度依赖训练数据中可用的上下文，这导致对以前未见过的场景的泛化能力较差。首选的高级表示应该在不失去自我意识的情况下捕获上下文信息。因此，我们建议保持具有相同身份但具有不同上下文的特征之间的上下文感知一致性，从而使表示对不同的环境具有鲁棒性。此外，我们提出了方向对比损失（DC Loss）以实现像素到像素的一致性，只需要将质量较低的特征与其对应部分对齐。此外，为了避免假阴性样本和过滤不确定的阳性样本，我们提出了两种采样策略。大量实验表明，我们简单而有效的方法大大超过了当前最先进的方法，并且还可以通过额外的图像级注释很好地概括。