深度学习 自动标记 图片_通过自动标记不确定性估计和主动学习来提高数据标记效率...

深度学习 自动标记 图片

机器学习 (Machine Learning)

In this post, we will be diving into the machine learning theory and techniques that were developed to evaluate our auto-labeling AI at Superb AI. More specifically, how our data platform estimates the uncertainty of auto-labeled annotations and applies it to active learning.

在本文中，我们将深入探讨在Superb AI上用来评估我们的自动标记AI的机器学习理论和技术。 更具体地说，我们的数据平台如何估计自动标记注释的不确定性并将其应用于主动学习。

Before jumping right in, it would be useful to have some mental buckets into which the most popular approaches can be categorized. In our experience, most works in deep learning uncertainty estimation fall under two buckets. The first belongs to the category of Monte-Carlo sampling, having multiple model inferences on each raw data and using the discrepancy between those to estimate the uncertainty. The second method models the probability distribution of model outputs by having a neural network learn the parameters of the distribution. The main intention here is to give breadth to the kind of techniques we explored and hope to offer some clarity on how and why we arrived at our unique position on the subject. We also hope to effectively demonstrate the scalability of our particular uncertainty estimation method.

在跳入之前，先将一些最流行的方法归类于其中，这将很有用。 根据我们的经验，深度学习不确定性估计中的大多数工作都归结为两个方面。 第一个属于蒙特卡洛抽样类别，对每个原始数据都有多个模型推论，并利用两者之间的差异来估计不确定性。 第二种方法是通过让神经网络学习分布的参数来对模型输出的概率分布进行建模 。 此处的主要目的是拓宽我们探索的技术种类，并希望就我们如何以及为什么在该主题上达到独特的位置提供一些清晰度。 我们也希望有效地证明我们特定不确定性估计方法的可扩展性。

1.快速回顾自动标记的功效 (1. A quick review of the efficacy of Auto-labeling)

Before we dive into the various approaches to evaluate the performance of auto labeling, there is a note of caution to be exercised. Auto-label AI, although extremely powerful, cannot always be 100% accurate. As such, we need to measure and evaluate how much we can trust the output when utilizing auto labeling. And once we can do it, the most efficient way to use auto-labeling is then to have a human user prioritize which auto-labeled annotation to review and edit based on this measure.

在我们深入研究评估自动标记性能的各种方法之前，需要注意一些注意事项。 自动标记的AI尽管功能非常强大，但并非总是100％准确。 因此，在使用自动标记时，我们需要测量和评估我们可以信任输出的程度。 并且一旦我们做到了，使用自动标记的最有效方法就是让人类用户根据此度量确定要检查和编辑的自动标记注释的优先级。

Measuring the “confidence” of model output is one popular method to do this. However, one well-known downside to this method is that confidence levels can be erroneously high even when the prediction turns out to be wrong if the model is overfitted to the given training data. Therefore, confidence levels cannot be used to measure how much we can “trust” auto-labeled annotations.

测量模型输出的“置信度”是一种常用的方法。 但是，该方法的一个众所周知的缺点是，即使模型过度适合给定的训练数据，即使预测结果是错误的，置信度也可能会错误地高。 因此，置信度不能用于衡量我们可以“信任”自动标记的注释多少。

In contrast, estimating the “uncertainty” of model output is a more grounded approach in the sense that this method statistically measures how much we can trust a model output. Using this, we can obtain an uncertainty measure that is proportional to the probability of model prediction error regardless of model confidence scores and model overfitting. This is why we believe that an effective auto-label technique needs to be coupled with a robust method to estimate prediction uncertainty.

相反，从某种意义上说，估计模型输出的“不确定性”是一种更扎实的方法，因为该方法从统计学上衡量了我们可以信任模型输出的程度。 使用此方法，我们可以获得与模型预测错误概率成比例的不确定性度量，而与模型置信度得分和模型过度拟合无关。 这就是为什么我们认为，有效的自动标记技术需要与可靠的方法相结合来估计预测不确定性的原因。

2.方法1：蒙特卡洛采样 (2. Method 1: Monte-Carlo Sampling)

One possible approach to uncertainty estimation proposed by the research community is obtaining multiple model outputs for each input data (i.e. images) and calculating the uncertainty using these outputs. This method can be viewed as a Monte-Carlo sampling-based method.

研究界提出的不确定性估计的一种可能方法是为每个输入数据(即图像)获得多个模型输出，并使用这些输出来计算不确定性。 该方法可以看作是基于蒙特卡洛采样的方法。

Let’s take a look at an example 3-class classification output below.

让我们看下面的示例3类分类输出。

y1 = [0.9, 0.1, 0]

y1 = [0.9，0.1，0]

y2 = [0.01, 0.99, 0]

y2 = [0.01，0.99，0]

y3 = [0, 0, 1]

y3 = [0，0，1]

y4 = [0, 0, 1]

y4 = [0，0，1]

Here, each of the y1 ~ y4 is the model output from four different models on the same input data (i.e. the first model gave the highest probability to class #1, etc.). The most naive approach would be using four different models to obtain these four outputs, but using Bayesian deep learning or dropout layers can give randomness to a single model and allow us to obtain multiple outputs from a single model.

在这里， y1〜y4中的