Lec2.1 Regularization

Lec2.1 Regularization

Most machine learning tasks are estimation of a function $\hat{f}(x)$ parameterized by a vector of parameters $\theta$.

A central problem in machine learning is how to make an algorithm that will perform well not just on the training data, but also on new inputs.

机器学习或深度学习会假设一个模型$\hat{f}(x)$，有一组参数$\theta$，然后根据训练数据，学习一组参数。我们想要这个模型不仅要在训练数据上表现好，还要在新的数据上表现好。

True function: $f(x)$, Estimate function $\hat{f}(x) $

Given a training set T = { ( x i , y i ) } i = 1 n T = \{(x_i, y_i)\}_{i=1}^{n} T={(xi​,yi​)}i=1n​， $y_i=f(x_i)+{ϵ}_i$，观测值，是真实值加上一个误差，${ϵ}_i\in N(0,{\sigma }^2)$.

对应一个给定的点，计算预测误差和真实误差的关系，如下。并且分两种情况，点在训练集中，和不在训练集中。

for point $(x_0,y_0)$,

$$\begin{gathered} E[({\hat{y}}_0-y_0{)}^2] \\ =E[({\hat{f}}_0-f_0-{ϵ}_0{)}^2] \\ =E[({\hat{f}}_0-f_0{)}^2]-2E[ϵ({\hat{f}}_0-f_0)]+{\sigma }^2 \end{gathered}$$

• Case 1, assume $(x_0,y_0)\notin T$

  $$\begin{gathered} E[{ϵ}_0({\hat{f}}_0-f_0)] \\ =E[(y_0-f_0)(\widehat{f_0}-f_0)] \\ =cov(y_0,\widehat{f_0}) \\ =0 \end{gathered}$$

  summing up over all m points that are not in T,

  $$\sum _{i=1}^m(\widehat{y_i}-y_i{)}^2=\sum _{i=1}^m(\widehat{f_i}-f_i{)}^2+m{\sigma }^2$$

  左侧是estimate error(err)，右侧第一项是true error(Err)，第二项是一个常数

  $Err=err-m{\sigma }^2$，这个等式说明，当点不在训练集中时，预测误差可以很好的代表实际误差，这就是交叉验证的原理。

• Case 2: assume $(x_0,y_0)\in T$

  插播一条定理，Stein’s Lemma，$x\in N(\theta ,{\sigma }^2),g(x)$可导，则$E[g(x)(x-\theta )]={\sigma }^2\frac{\partial g(x)}{\partial x}$

  $$\begin{gathered} E[{ϵ}_0(\widehat{f_0}-f_0)] \\ ={\sigma }^2\frac{\partial (\widehat{f_0}-f_0)}{\partial {ϵ}_0} \\ ={\sigma }^2\frac{\partial \widehat{f_0}}{\partial y_0}\frac{\partial y_0}{\partial {ϵ}_0} \\ ={\sigma }^2\frac{\partial \widehat{f_0}}{\partial y_0} \\ ={\sigma }^2{D}_0 \end{gathered}$$

  $D_0$代表模型的复杂程度。

  summing up over all m points that are in T,

  $$\sum _{i=1}^m(\widehat{y_i}-y_i{)}^2=\sum _{i=1}^m(\widehat{f_i}-f_i{)}^2-2{\sigma }^2\sum _{i=1}^m{D}_i+m{\sigma }^2$$

  训练误差(err)不能代表真实误差(Err), true error is estimate error plus a bias.

  $Err=err+2{\sigma }^2{\sum }_{i=1}^m{D}_i-m{\sigma }^2$.

  这就是正则化Regularization的原因 $J(\theta ;x,y)+\Omega (\theta )$，在训练误差的基础上加上一个与模型复杂度相关的偏置。
  theta;x,y)+\Omega(\theta)$，在训练误差的基础上加上一个与模型复杂度相关的偏置。
  这里可以看上图中的函数图，随着模型复杂度的增加，训练误差逐渐减少，真实误差，先减少后增大。