Coursera | Andrew Ng (03-week1-1.9)—可避免偏差

该系列仅在原课程基础上部分知识点添加个人学习笔记，或相关推导补充等。如有错误，还请批评指教。在学习了 Andrew Ng 课程的基础上，为了更方便的查阅复习，将其整理成文字。因本人一直在学习英语，所以该系列以英文为主，同时也建议读者以英文为主，中文辅助，以便后期进阶时，为学习相关领域的学术论文做铺垫。- ZJ

Coursera 课程 |deeplearning.ai |网易云课堂

---

转载请注明作者和出处：ZJ 微信公众号-「SelfImprovementLab」

知乎：https://zhuanlan.zhihu.com/c_147249273

CSDN：http://blog.csdn.net/junjun_zhao/article/details/79157399

---

1.9 Avoidable bias (可避免偏差)

(字幕来源：网易云课堂)

We talked about how you want your learning algorithm to do well on the training set but sometimes you don’t actually want to do too well and knowing what human level performance is,can tell you exactly how well but not too well you want your algorithm to do on the training set.Let me show you what I mean.We have used Cat classification a lot and giving a picture,let’s say humans have near-perfect accuracy so the human level error is one percent.In that case, if your learning algorithm achieves 8 percent training error and 10 percent dev error,then maybe you wanted to do better on the training set.So the fact that there’s a huge gap between how well your algorithm does on your training set versus how humans do shows that your algorithm isn’t even fitting the training set well.So in terms of tools to reduce bias or variance,in this case I would say focus on reducing bias.So you want to do things like train a bigger neural network or run gradient descent longer,just try to do better on the training set.

我们讨论过 你希望你的学习算法能在训练集上表现良好，但有时你实际上并不想做得太好，你得知道人类水平的表现是怎样的，可以确切告诉你 算法在训练集上的表现，到底应该有多好 或者有多不好，让我告诉你是什么意思吧，我们经常使用猫分类器来做例子，比如人类具有近乎完美的准确度 所以人类水平的错误是百分之一，在这种情况下 如果您的学习算法达到 8 % 的训练错误率和 10% 的开发错误率，那么你也许想在训练集上得到更好的结果，所以事实上 你的算法在训练集上的表现和人类水平的表现有很大差距的话, 说明你的算法对训练集的拟合并不好，所以从减少偏差和方差的工具这个角度看，在这种情况下 我会把重点放在减少偏差上，你需要做的是 比如说训练更大的神经网络 或者跑久一点梯度下降，就试试能不能在训练集上做得更好。

But now let’s look at the same training error and dev error and imagine that human level performance was not 1%.So just copy this over but you know in a different application or maybe on a different data set,let’s say that human level error is actually 7.5%.Maybe the images in your data set are so blurry that even humans can’t tell whether there’s a cat in this picture.This example is maybe slightly contrived because humans are actually very good at looking at pictures and telling if there’s a cat in it or not.But for the sake of this example,let’s say your data sets images are so blurry or so low resolution that even humans get 7.5% error.In this case, even though your training error and dev error are the same as the other example,you see that maybe you’re actually doing just fine on the training set.It’s doing only a little bit worse than human level performance.And in this second example,you would maybe want to focus on reducing this component,reducing the variance in your learning algorithm.So you might try regularization to try to bring your dev error closer to your training error for example.So in the earlier courses discussion on bias and variance,we were mainly assuming that there were tasks where Bayes error is nearly zero.

但现在我们看看同样的训练错误率和开发错误率，假设人类的表现不是1%，我们就把它抄写过来，但你知道 在不同的应用 或者说用在不同的数据集上，假设人类水平错误实际上是 7.5%。，也许你的数据集中的图像非常模糊 即使人类，都无法判断这张照片中有没有猫，这个例子可能稍微更复杂一些，因为人类其实很擅长看照片，分辨出照片里有没有猫，但就为了举这个例子，比如说你的数据集中的图像非常模糊 分辨率很低，即使人类错误率也达到 7.5%，在这种情况下 即使，你的训练错误率和开发错误率和其他例子里一样，你就知道 也许你的系统在训练集上的表现还好，它只是比人类的表现差一点点，在第二个例子中，你可能希望专注减少这个分量，减少学习算法的方差，也许你可以试试正则化，让你的开发错误率更接近你的训练错误率，所以在之前的课程关于偏差和方差的讨论中，我们主要假设 有一些任务的贝叶斯错误率几乎为 0。

So to explain what just happened here,for our Cat classification example,think of human level error as a proxy or as a estimate for Bayes error or for Bayes optimal error.And for computer vision tasks,this is a pretty reasonable proxy because humans are actually very good at computer vision and so whatever a human can do is maybe not too far from Bayes error.By definition, human level error is worse than Bayes error because nothing could be better than Bayes error but human level error might not be too far from Bayes error.So the surprising thing we saw here is that depending on what human level error is or really this is really approximately Bayes error or so we assume it to be,but depending on what we think is achievable,with the same training error and dev error in these two cases,we decided to focus on bias reduction tactics or on variance reduction tactics.And what happened is in the example on the left, 8% training error is really high when you think you could get it down to 1%and so bias reduction tactics could help you do that.Whereas in the example on the right,if you think that Bayes error is 7.5%and here we’re using human level error as an estimate or as a proxy for Bayes error,but you think that Bayes error is close to 7.5% then you know there’s not that much headroom for reducing your training error further down.You don’t really want it to be that much better than 7.5% because you could achieve that only by maybe starting to offer further training so,and instead, there’s much more room for improvement in terms of taking this 2% gap and trying to reduce that by using variance reduction techniques such as regularization or maybe getting more training data.

所以要解释这里发生的事情，看看这个猫分类器，用人类水平的错误率，估计或代替贝叶斯错误率 或贝叶斯最优错误率，对于计算机视觉任务而言，这样替代相当合理 因为人类实际上是非常擅长，计算机视觉任务的 所以人类能做到的水平 和贝叶斯错误率相差不远，根据定义 人类水平错误率比贝叶斯错误率高一点，因为贝叶斯错误率是理论上限，但人类水平错误率离贝叶斯错误率不会太远，所以这里比较意外的是 取决于人类水平错误率有多少，或者这真的就很接近贝叶斯错误率 所以我们假设它就是，但取决于我们认为什么样的水平是可以实现的，在这两种情况下 具有同样的训练错误率和开发错误率，我们决定专注于减少偏差的策略 或者减少方差的策略，那么左边的例子发生了什么? 8% 的训练错误率真的很高 你认为你可以把它降到 1%，那么减少偏差的手段可能有效，而在右边的例子中，如果你认为贝叶斯错误率是 7.5%，这里我们使用人类水平错误率来替代贝叶斯错误率，但是你认为贝叶斯错误率接近 7.5%，你就知道没有太多改善的空间了，不能继续减少你的训练错误率了，你也不会希望它比 7.5% 好得多，因为这种目标只能通过，可能需要提供更进一步的训练，而这边 就还有更多的改进空间，可以将这个 2% 的差距缩小一点，使用减少方差的手段应该可行，比如正则化 或者收集更多的训练数据。

So to give these things a couple of names,this is not widely used terminology but I found this useful terminology and a useful way of thinking about it,which is I’m going to call the difference between Bayes error or approximation of Bayes error and the training error to be the avoidable bias.So what you want is maybe keep improving your training performance until you get down to Bayes error but you don’t actually want to do better than Bayes error.You can’t actually do better than Bayes error unless you’re overfitting.And this, the difference between your training area and the dev error,there’s a measure still of the variance problem of your algorithm.And the term avoidable bias acknowledges that there’s some bias or some minimum level of error that you just cannot get below which is that if Bayes error is 7.5%,you don’t actually want to get below that level of error.So rather than saying that if you’re training error is 8%,then the 8% is a measure of bias in this example,you’re saying that the avoidable bias is maybe 0.5% or 0.5% is a measure of the avoidable bias whereas 2% is a measure of the variance and so there’s much more room in reducing this 2% than in reducing this 0.5%.Whereas in contrast in the example on the left,this 7% is a measure of the avoidable bias,whereas 2% is a measure of how much variance you have.And so in this example on the left,there’s much more potential in focusing on reducing that avoidable bias.

所以要给这些概念命名一下，这不是广泛使用的术语 但我觉得这么说，思考起来比较流畅，就是把这个差值，贝叶斯错误率或者对贝叶斯错误率的估计 和训练错误率之间的差值，称为可避免偏差，你可能希望一直提高训练集表现，直到你接近贝叶斯错误率，但实际上你也不希望做到比贝叶斯错误率更好，这理论上是不可能超过贝叶斯错误率的 除非过拟合，而这个 训练错误率和开发错误率之前的差值，就大概说明你的算法在方差问题上还有多少改善空间，可避免偏差这个词 说明了，有一些别的偏差 或者错误率有个无法超越的最低水平，那就是说如果贝叶斯错误率是 7.5%，你实际上并不想得到低于该级别的错误率，所以你不会说你的训练错误率是 8%，然后 8% 就衡量了例子中的偏差大小，你应该说 可避免偏差可能在 0.5% 左右，或者 0.5% 是可避免偏差的指标，而这个 2% 是方差的指标，所以要减少这个 2% 比减少这个 0.5% 空间要大得多，而在左边的例子中，这 7% 衡量了可避免偏差大小，而 2% 衡量了方差大小。

So in this example,understanding human level error,understanding your estimate of Bayes error really causes you in different scenarios to focus on different tactics,whether bias avoidance tactics or variance avoidance tactics.There’s quite a lot more nuance in how you factor in human level performance into how you make decisions in choosing what to focus on.Thus in the next video,go deeper into understanding of what human level performance really mean.

所以在左边这个例子里，专注减少可避免偏差可能潜力更大，所以在这个例子中，当你理解人类水平错误率，理解你对贝叶斯错误率的估计，你就可以在不同的场景中专注于不同的策略，使用避免偏差策略还是避免方差策略，在训练时如何考虑人类水平表现 来决定工作着力点，具体怎么做还有更多微妙的细节，所以在下一个视频中，我们会深入了解人类水平表现的真正意义。

---

重点总结：

与人类表现做比较

可避免偏差

假设针对两个问题分别具有相同的训练误差和交叉验证误差，如下所示：

对于左边的问题，人类的误差为 1%，对于右边的问题，人类的误差为 7.5%。

对于某些任务如计算机视觉上，人类能够做到的水平和贝叶斯误差相差不远。（这里贝叶斯误差指最好的分类器的分类误差，也就是说没有分类器可以做到100% 正确）。这里将人类水平误差近似为贝叶斯误差。

• 左边的例子：8% 与 1% 差距较大
  主要专注于减少偏差，即减少训练集误差和人类水平误差之间的差距，来提高模型性能。
• 右边的例子：8% 与 7.5% 接近
  主要专注于减少方差，即减少开发集误差和测试集误差之间的差距，来提高模型性能。

参考文献：

[1]. 大树先生.吴恩达Coursera深度学习课程 DeepLearning.ai 提炼笔记（3-1）– 机器学习策略（1）

---

PS: 欢迎扫码关注公众号：「SelfImprovementLab」！专注「深度学习」，「机器学习」，「人工智能」。以及 「早起」，「阅读」，「运动」，「英语 」「其他」不定期建群 打卡互助活动。