神经网络 对话_我确定或不确定与神经网络的对话

神经网络 对话

If your cell phone is suggesting whether a mushroom you just picked up from your backyard for your dinner is poisonous, it better be sure about it! Computer vision applications are now dominated by the applications of neural networks. They have affected each and every one of our lives, thus, also increasing the suspicion. There have been many debates on the trustworthiness of this black box machine. One way by which a model can make itself more trustworthy is by telling us how sure or unsure is it of its decision rather than just saying ‘this is my decision’.

如果您的手机提示您刚从后院拿起的晚餐中食用的蘑菇是否有毒，那么最好确定一下！ 现在，计算机视觉应用主要由神经网络应用主导。 他们影响了我们的每一生，因此也增加了人们的怀疑。 关于这种黑匣子机器的可信赖性已有许多争论 。 模型可以使自己更加值得信赖的一种方法是，告诉我们其决策的确定性或不确定性，而不仅仅是说“这是我的决策”。

But then the question comes down to how good the current state of the art neural networks are at determining how sure they are of their decision.

但是接下来的问题归结为，目前最先进的神经网络在确定他们的决策确定性方面有多出色。

Through this post, we will understand —

通过这篇文章，我们将了解-

• What predicting certainty means for a model [Confidence Calibration]
  预测确定性对模型意味着什么[置信度校准]
• How can we measure it? [Reliability diagram and ECE]
  我们如何衡量呢？ [可靠性图和ECE]
• Why should we expect a model to be calibrated in the very first place? [MAP, KL divergence and cross entropy]
  我们为什么要首先对模型进行校准？ [MAP，KL散度和交叉熵]

让我们开始旅程-置信度校准 (Let's start our journey - Confidence Calibration)

The problem of predicting probability estimates representative of the true likelihood is called confidence calibration. For example, given 100 predictions, each with the confidence of 0.8, we expect that 80 should be correctly classified. The easiest way to measure this is a plot of confidence vs accuracy. This is known as the Reliability Diagram.

预测代表真实可能性的概率估计值的问题称为置信度校准。 例如，给定100个预测，每个预测的置信度为0.8，我们预计应正确分类80个。 衡量这一点的最简单方法是置信度与准确性的关系图。 这就是所谓的可靠性图 。

The x-axis here is the Bin Accuracy or number of correct samples/ total number of samples in that bin-

这里的x轴是Bin准确度或正确样本数/该bin-中的样本总数

Bin accuracy

料箱精度

and the y-axis is the confidence or the average of the probabilities in that bin -

y轴是该区间中的置信度或概率的平均值-

Confidence Calibration

置信度校准

If the confidence always equals the accuracy i.e the x=y line the model is perfectly calibrated. If a model’s calibration plot lies above the x=y (Perfect Calibration)line this means it gives higher probability estimates for correct classes which makes it overconfident. Lower probability estimates for correct classes means the model is underconfident.

如果置信度始终等于精度，即x = y线，则表示模型已完美校准。 如果模型的校准图位于x = y(完美校准)线以上，则意味着对正确的类别给出了更高的概率估计，这使其过于自信 。 正确分类的较低概率估计值意味着该模型不可靠。

校准分数—预期校准误差(ECE) (The score for Calibration — Expected Calibration Error (ECE))

Can we convert the reliability diagram into a number for comparing the calibrations of different models?

我们可以将可靠性图转换为用于比较不同模型的校准的数字吗？

The difference in expectation between confidence and accuracy

置信度和准确性之间的期望差异

Intuitively it can be understood as the difference between the model’s calibration plot and the x=y line i.e perfect calibration line.

直观上可以理解为模型的校准图与x = y线(即完美校准线)之间的差异。

真实交易-为什么我们应该期望模型首先被校准？ (Real Deal — Why should we expect a model to be calibrated in the very first place?)

We optimize a neural network by maximum likelihood estimation (MLE) given by

我们通过下式给出的最大似然估计( MLE )优化神经网络

i.e we maximize the probability of data given the weights/ model parameters. If we add a regulariser term we can see MLE converts to a maximum a posteriori (MAP) estimate. P(w) is l2 regularization if we have a Gaussian prior over W and l1 regularization if we have a Laplace prior over W.

即在给定权重/模型参数的情况下，我们最大化数据的概率。 如果添加正则项，我们可以看到MLE转换为最大后验( MAP )估计。 如果我们在W上有一个高斯先验，则P(w)是l2正则化;如果我们在W上有一个Laplace先验，则P(w)是l1正则化。

We want the probability output by the model to match the true posterior of the data! We need a loss function which minimizes the distance between the distribution over labels output by the model to the exact posterior distribution of the data (we minimize the distance between two distributions by KL-divergence).

我们希望模型输出的概率与数据的真实后验相匹配！ 我们需要一个损失函数，以最小化模型输出的标签到数据的后验分布之间的距离(通过KL散度最小化两个分布之间的距离)。

KL散度和损失函数 (KL Divergence and Loss function)

First, let's see what is usually ‘taken’ as the ‘posterior’ in the case of single-class classification looks. If we have two classes — a cat and dog (please note this nowhere represents the true posterior of the data). The posterior, if a dog is present would look like [0,1] and if a cat is present is would be [1,0]. (Our posterior looks like a delta function in this case)

首先，让我们看一下在单类分类的情况下通常被称为“后验”的东西。 如果我们有两类-猫和狗(请注意，这无处代表数据的真实后验)。 如果有狗，则后方看起来像[0,1]，如果有猫，则后方看起来是[1,0]。 (在这种情况下，我们的后眼看起来像是三角函数 )

The KL-divergence for this binary classification task can be given by -

此二进制分类任务的KL散度可通过-

Where x is the input picture, P is the actual distribution and P_theta is the computed by our model.

其中x是输入图片，P是实际分布，P_theta是我们的模型计算的。

The average KL divergence can be given by —

KL的平均散度可通过以下公式得出：

We can already see this taking the form of the most commonly used loss function to optimize neural networks - the cross-entropy.

我们已经可以看到它采用最常用的损失函数形式来优化神经网络- 交叉熵 。

Further simplifying —

进一步简化—

Since the first term does not depend on theta term and therefore does not affect the argmin_theta. The resulting equation is the cross-entropy loss —

由于第一项不依赖于theta项，因此不影响argmin_theta。 结果方程式是交叉熵损失-

Therefore minimizing the cross-entropy is equivalent to minimizing the KL-divergence between model output and posterior from the data. Thus theoretically any neural network should be perfectly calibrated (There are many causes which lead to miscalibration, let's keep it for another post for now!)

因此，最小化交叉熵等效于最小化模型输出与数据后验之间的KL散度。 因此，理论上任何神经网络都应该进行完美校准(有许多原因会导致校准错误，让我们暂时将其保留在另一篇文章中！)

[1] pymc https://docs.pymc.io/notebooks/bayesian_neural_network_advi.html

[1] pymc https://docs.pymc.io/notebooks/bayesian_neural_network_advi.html

[2] On Calibration of Neural Networks https://arxiv.org/pdf/1706.04599.pdf

[2]关于神经网络的校准https://arxiv.org/pdf/1706.04599.pdf

翻译自: https://towardsdatascience.com/am-i-sure-or-unsure-talks-with-a-neural-network-fc0e14d31373

神经网络 对话