Introduction
在 ESL和 ISLR中,都给出了对于 bias和 variance的讨论,并给出这样的结论:
E r r ( X ) = V a r ( f ^ ( X ) ) + B i a s ( f ^ ( X ) ) 2 + V a r ( ϵ ) Err(X) = Var(\hat{f}(X)) + Bias(\hat{f}(X))^2 + Var(\epsilon) Err(X)=Var(f^(X))+Bias(f^(X))2+Var(ϵ)
但是笔者在查阅资料时发现,对这个结论的少有比较详尽的推导。故在此整合后加上自己的理解,将一个比较详尽的推导过程给出。
在训练一个模型的时候,我们最关心的就是模型的 Bias和 Variance。假设真实的数据由函数和noise(通常假设为高斯噪音)给出:
Y
=
f
(
X
)
+
ϵ
Y = f(X) + \epsilon
Y=f(X)+ϵ
通过数据,我们学习到了一个模型
f
^
(
X
)
\hat{f}(X)
f^(X),那么,这个模型的准确如何评估呢?
E
r
r
(
X
)
:
=
E
[
(
Y
−
f
^
(
X
)
)
2
]
Err(X) := E[(Y-\hat{f}(X))^2]
Err(X):=E[(Y−f^(X))2]
可以看出,MSE其实就是这个式子的一个实际估算值。而将这个式子展开,我们可以发现:天下没有免费的午餐,bias和 variance几乎一定是不可兼得滴~
Derivation
E [ ( Y − f ^ ( X ) ) 2 ] = E [ ( f ( X ) + ϵ − f ^ ( X ) ) 2 ] = E { [ ( f ^ ( X ) − f ( X ) ) + ϵ ] 2 } = E { ( f ^ ( X ) − f ( X ) ) 2 + 2 ϵ ( f ^ ( X ) − f ( X ) ) + ϵ 2 } E[(Y-\hat{f}(X))^2] = E[(f(X) + \epsilon - \hat{f}(X))^2] = E\{ [ (\hat{f}(X) - f(X)) + \epsilon ]^2 \} = E\{ (\hat{f}(X)-f(X))^2 + 2\epsilon (\hat{f}(X) - f(X)) + \epsilon^2 \} E[(Y−f^(X))2]=E[(f(X)+ϵ−f^(X))2]=E{[(f^(X)−f(X))+ϵ]2}=E{(f^(X)−f(X))2+2ϵ(f^(X)−f(X))+ϵ2}
因为E是线性的,所以进一步得到: E [ ( f ^ ( X ) − f ( X ) ) 2 ] + 2 E [ ϵ ( f ^ ( X ) − f ( X ) ) ] + E [ ϵ 2 ] E[(\hat{f}(X)-f(X))^2] + 2E[\epsilon (\hat{f}(X) - f(X))] + E[\epsilon^2] E[(f^(X)−f(X))2]+2E[ϵ(f^(X)−f(X))]+E[ϵ2]
注意到 ϵ \epsilon ϵ 是一个噪声(通常来自测量误差),因此它和 X X X是独立的,故有: E [ ϵ ( f ^ ( X ) − f ( X ) ) ] = E [ ϵ ] E [ ( f ^ ( X ) − f ( X ) ) ] = 0 E[\epsilon (\hat{f}(X) - f(X))] = E[\epsilon] E[(\hat{f}(X) - f(X))] = 0 E[ϵ(f^(X)−f(X))]=E[ϵ]E[(f^(X)−f(X))]=0
再注意到 V a r ( ϵ ) = E [ ϵ 2 ] − [ E ( ϵ ) ] 2 = E [ ϵ 2 ] Var(\epsilon) = E[\epsilon^2] - [E(\epsilon)]^2 = E[\epsilon^2] Var(ϵ)=E[ϵ2]−[E(ϵ)]2=E[ϵ2]
这样我们就得到了 E [ ( Y − f ^ ( X ) ) 2 ] = E [ ( f ^ ( X ) − f ( X ) ) 2 ] + V a r ( ϵ ) E[(Y-\hat{f}(X))^2] = E[(\hat{f}(X)-f(X))^2] + Var(\epsilon) E[(Y−f^(X))2]=E[(f^(X)−f(X))2]+Var(ϵ)
最后,我们要做的是将第一项展开,
E [ ( f ^ ( X ) − f ( X ) ) 2 ] = E [ ( f ^ ( X ) − E ( f ^ ( X ) ) + E ( f ^ ( X ) ) − f ( X ) ) 2 ] = E [ ( f ^ ( X ) − E ( f ^ ( X ) ) ) 2 ] + 2 E { [ f ^ ( X ) − E ( f ^ ( X ) ) ] [ E ( f ^ ( X ) ) − f ( X ) ] } + E [ ( E ( f ^ ( X ) ) − f ( X ) ) 2 ] E[(\hat{f}(X)-f(X))^2] = E[(\hat{f}(X) - E(\hat{f}(X)) + E(\hat{f}(X)) -f(X))^2] \\ = E[(\hat{f}(X) - E(\hat{f}(X)))^2] + 2 E\{ [ \hat{f}(X) - E(\hat{f}(X)) ] [ E(\hat{f}(X)) -f(X) ]\} + E[(E(\hat{f}(X)) -f(X))^2] E[(f^(X)−f(X))2]=E[(f^(X)−E(f^(X))+E(f^(X))−f(X))2]=E[(f^(X)−E(f^(X)))2]+2E{[f^(X)−E(f^(X))][E(f^(X))−f(X)]}+E[(E(f^(X))−f(X))2]
注意到, E ( f ^ ( X ) ) − f ( X ) E(\hat{f}(X)) - f(X) E(f^(X))−f(X) 事实上是一个常数,与E无关,因此 E { [ f ^ ( X ) − E ( f ^ ( X ) ) ] [ E ( f ^ ( X ) ) − f ( X ) ] } = [ E ( f ^ ( X ) ) − f ( X ) ] E [ f ^ ( X ) − E ( f ^ ( X ) ) ] = 0 E\{ [ \hat{f}(X) - E(\hat{f}(X)) ] [ E(\hat{f}(X)) -f(X) ]\} = [E(\hat{f}(X)) - f(X)] E[ \hat{f}(X) - E(\hat{f}(X)) ] = 0 E{[f^(X)−E(f^(X))][E(f^(X))−f(X)]}=[E(f^(X))−f(X)]E[f^(X)−E(f^(X))]=0
What’s left?
E [ ( f ^ ( X ) − E ( f ^ ( X ) ) ) 2 ] = V a r ( f ^ ( X ) ) E[(\hat{f}(X) - E(\hat{f}(X)))^2] = Var(\hat{f}(X)) E[(f^(X)−E(f^(X)))2]=Var(f^(X))
E [ ( E ( f ^ ( X ) ) − f ( X ) ) 2 ] = ( E ( f ^ ( X ) ) − f ( X ) ) 2 = B i a s ( f ^ ( X ) ) 2 E[(E(\hat{f}(X)) -f(X))^2] = (E(\hat{f}(X)) -f(X))^2 = Bias(\hat{f}(X))^2 E[(E(f^(X))−f(X))2]=(E(f^(X))−f(X))2=Bias(f^(X))2
因此,我们有了最终的结论: E r r ( X ) = V a r ( f ^ ( X ) ) + B i a s ( f ^ ( X ) ) 2 + V a r ( ϵ ) Err(X) = Var(\hat{f}(X)) + Bias(\hat{f}(X))^2 + Var(\epsilon) Err(X)=Var(f^(X))+Bias(f^(X))2+Var(ϵ)
Discussion
从这个式子我们可以直观地看出:给定数据集, E r r ( X ) Err(X) Err(X) 是不会改变的(因此,多收集一些数据或许可以使得它减小)。而 V a r ( ϵ ) Var(\epsilon) Var(ϵ) 更是难以改进。因此,Bias 和 Variance的 trade-off就是一个永恒的难题。或许,这也正是 data science的魅力所在吧。
References:
https://liam.page/2017/03/25/bias-variance-tradeoff/
https://stats.stackexchange.com/questions/204115/understanding-bias-variance-tradeoff-derivation