【论文阅读】Label Encoding for Regression Networks

论文下载
bib:

@INPROCEEDINGS{DevalZiYu2022Label,
  author    = {Deval Shah and Zi Yu Xue and Tor M. Aamodt},
  title     = {Label Encoding for Regression Networks},
  booktitle = {ICLR},
  year      = {2022},
  pages     = {1--35}
}

1. 摘要

1. Deep neural networks are used for a wide range of regression problems. However, there exists a significant gap in accuracy between specialized approaches and generic direct regression in which a network is trained by minimizing the squared or absolute error of output labels.

note: 论文八股的表现，描述现状，however引出问题。

2. Prior work has shown that solving a regression problem with a set of binary classifiers can improve accuracy by utilizing well-studied binary classification algorithms.

note: 前人工作，引出将回归问题转化为一组二分类问题的解决方案，由前人工作引出这是有效的。

3. We introduce binary-encoded labels (BEL), which generalizes the application of binary classification to regression by providing a framework for considering arbitrary multi-bit values when encoding target values. We identify desirable properties of suitable encoding and decoding functions used for the conversion between real-valued and binary-encoded labels based on theoretical and empirical study. These properties highlight a tradeoff between classification error probability and error-correction capabilities of label encodings. BEL can be combined with off-the-shelf task-specific feature extractors and trained end-to-end. We propose a series of sample encoding, decoding, and training loss functions for BEL and demonstrate they result in lower error than direct regression and specialized approaches while being suitable for a diverse set of regression problems, network architectures, and evaluation metrics.

note: 本文的主要工作，也就是写在In this paper之后的内容。从摘要中可以看出，本文主要是提出一个框架，相比于直接处理回归问题，本文将回归通过编码器、解码器转化为一组二分类问题来解决。其实感觉创新点不够硬，因为这样处理回归问题已经出现过了，但是可以看出工作量惊人，较为系统的分析了整个框架，消融实验，对比实验是做的真的多。

2. 算法描述

Figure 1 很清晰的表述了本文提出处理回归问题的框架。其中，红色的部分表示本文集中的部分，有四个部分。

1. Quantization： 可以看作是一个缩放加取整的操作。其中，也有非均匀量化（nonuniform quantization）的方法，在本文中采用的是均匀量化。
2. Encoding：编码器。这个是核心的组件，将回归问题的实数标签转化为一组二进制离散标签。
3. Training binary-classifiers: 训练一组二分类模型。
4. Decoding： 解码器。将二分类器输出的离散二进制编码转化为实数回归标签。这个部分很重要，涉及到优化目标以及损失函数的设计。其中，是最小化离散标签的距离，还是解码实数标签的距离，作者也对此做了对比实验。

3. 选择编码器的经验方法

1. individual classifiers should require fewer bit transitions as this makes them easier to train.

note: 编码器输出编码应该尽可能的短。

2. a desirable property for a BEL encoding function is that the hamming distance between two codes (number of bits that differ) should be proportional to the difference between the target values they encode.

note: 编码器输出编码应该尽可能的短。Hamming Distance表示两个等长序列中不同位数的个数。第二点就是说编码器输出的每一位应该是同等权重，应为每一位编码代表一个二分类器，它们享有同等的犯错权力。

3. For a given target value classifiers closer to a bit transition are more likely to incur an error.

note: 这个说的是与目标值越接近越容易犯错。这个说的是边界难以区分，这也是训练后期波动的主要原因。

4. 总结

本文的核心就是将回归问题有效的转化为一组二分类问题。其中，这样的做法之前已经出现过，不算是核心的创新，但是作者对于多回归任务，多框架组件组合来较为系统的分析了提出框架的有效性，并对此做了很多的消融和对比实验，工作量很大。