Coursera ML笔记 -----week3 Regulariztion

欢迎点击作者原文地址

2017/3/20

Regularization

正则化的目的是改善或者减少过拟合overfitting

1. 什么是overfitting？
   underfitting /high bias欠拟合,是指 not fitting training set very well
   overfitting /high variance 过拟合，是指 too many features,fit the training set very well($J(\theta)\approx 0$),but fail to generalize to new examples.

2. 怎么去解决过拟合的问题

   • reduce the num of features (但是可能会丢失一部分信息)
   • regularization

3. penalize parameters
   在原来的cost function上面增加一个regularize term
   

   $$J(\theta)=J(\theta)+\frac{\lambda}{2m}\sum_{j=1}^m\theta_j^2$$ ,于是我们将这一项分别应用到线性回归的Gradient descent 和Normal Equation 和逻辑回归里面去

4. regularize linear regression

   • 梯度下降
     repeat{
     

     $$\theta_0 := \theta_0 - \alpha \frac{1}{m}\sum_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})^2x_0^{(i)}$$
     $$\theta_j := \theta_j - \alpha \frac{1}{m}\left[\sum_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})^2x_j^{(i)}+\lambda\theta_j\right]=\theta_j(1-\alpha\frac{\lambda}{m}) - \alpha \frac{1}{m}\sum_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})^2x_j^{(i)}$$
     $j=1,2,3,...,n$
     }

   • Normal Equation
     

     $$\theta =\left( X^{T}X+\lambda\begin{bmatrix} 0\\ & 1\\ &&&\ddots\\ &&&&1 \end{bmatrix}\right)^{-1}X^{T}y$$

5. regularized logistic regression

   • 梯度下降，和线性回归的形式是一样的，但是并不是同一个函数，因为$h_\theta(x)$并不相同

   repeat{
   

   $$\theta_0 := \theta_0 - \alpha \frac{1}{m}\sum_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})^2x_0^{(i)}$$
   $$\theta_j := \theta_j - \alpha \frac{1}{m}\left[\sum_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})^2x_j^{(i)}+\lambda\theta_j\right]=\theta_j(1-\alpha\frac{\lambda}{m}) - \alpha \frac{1}{m}\sum_{i=1}^m(h_\theta(x^{(i)})-y^{(i)})^2x_j^{(i)}$$
   $j=1,2,3,...,n$
   }