bias 和 variance 的推导

参考 博客ooon

假设 $X和Y 满足Y=X+\varepsilon$，$\varepsilon$是噪声，$\varepsilon \thicksim N(0,\delta_{\varepsilon}^2)$。
$f(X)是准确的预测函数，\hat{f}(X)是通过训练集训练的预测函数。$

$$\begin{align} Err(x_0)&=E[(y_0-\hat{f}(x_0))^2] \\ &=E[y_0^2-2y_0\hat{f}(x_0)+\hat{f}^2(x_0)] \\ &=E[y_0^2]-2E[y_0\hat{f}(x_0)]+E[\hat{f}^2(x_0)] \\ &=E[y_0^2]-E^2(y_0)+E^2(y_0)-2E[y_0\hat{f}(x_0)]+E[\hat{f}^2(x_0)] -E^2(\hat{f}(x_0))+E^2(\hat{f}(x_0))\\ \end{align}$$
$$\begin{align} \because \ &E[y_0]=E[f(x_0+\varepsilon)]=E(f(x_0))\\ &Var[y_0]=E[(y_0-E[y_0])^2]=E[(y_0+\varepsilon-y_0)^2]=E[\varepsilon^2]\\ &Var[X]=E[X^2]-E^2[X] \end{align}$$

$$\begin{align} \therefore \ Err(x_0)&=[E[y_0]-E[\hat{f}(x_0)]]^2+E[\hat{f}^2(x_0)]-E^2(\hat{f}(x_0))+Var[\varepsilon]\\ &=[E[y_0]-E[\hat{f}(x_0)]]^2+Var[\hat{f}(x_0)]+Var[\varepsilon] \\ &=Bias^2(\hat{f}(x_0))+Var[\hat{f}(x_0)]+\delta_\varepsilon^2 \end{align}$$

对于k近邻，其预测误差为:

$$\begin{align} Err(x_0)&=E[(Y-\hat{f}(x_0))|X=x_0] \\ &=\delta_\varepsilon^2+[f(x_0)-\frac{1}{k}\sum^K_{i=1}f(x_i)]^2+\frac{\delta_\epsilon^2}{k} \end{align}$$

随着 $k$ 增大，$var$ 会减小，而$bias$ 会增大。