Notes on Machine Learning By Andrew Ng (4)

Notes on Machine Learning By Andrew Ng (4)

Click here to check former notes.

Regularization（正则化）

The problem of overfitting

Addressiong overfitting

Options:

• Reduce # of features
• • Manually select which features to keep.
• • Model selection algorithm (later in course) .
• • abandon some useful info.
• Regularization
• • Keep all the features, but reduce magnitude/ values of parameters ${\theta }_j$.
• • Works well when we have a lot of features, each of which contributes a bit to predicting $y$.

Cost Function

Regularization

Small values for parameters ${\theta }_i,1\le i\le n$

• “Simpler” hypothesis
• Less prone to overfitting

$J(\theta )=\frac{1}{2m}[{\sum }_{i=1}^m{(h_{\theta }(x^{(i)})-y^{(i)})}^2+\lambda {\sum }_{i=1}^n{\theta }_i^2]$

Regularzation term: $\lambda {\sum }_{i=1}^n{\theta }_i^2$

Regularzation parameter: $\lambda$, controls a trade off between 2 different goals.

• First goal captured by the first term of the objective, (a.k.a ${\sum }_{i=1}^m{(h_{\theta }(x^{(i)})-y^{(i)})}^2$), to fit the data well.
• Second goal is keep the parameters small, so that avoid overfitting.

If $\lambda$ is set too large, it will be like ${\theta }_i\approx 0,1\le i\le n$, which will lead to underfitting.

Regularized linear regression

Gradient descent

Modify its process.

Repeat{

${\theta }_0:={\theta }_0-\alpha \frac{1}{m}{\sum }_{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_0^{(i)}$

${\theta }_j:={\theta }_j-\alpha [\frac{1}{m}{\sum }_{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}+\frac{\lambda }{m}{\theta }_j](j=1,2\cdots ,n)$

}

$\to {\theta }_j:={\theta }_j(1-\alpha \frac{\lambda }{m})-\alpha \frac{1}{m}{\sum }_{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}$

It is a fact that $1-\alpha \frac{\lambda }{m}<1$, so $\theta$ descent a little in every progress.

Normal equation

Modify it.
$$\theta =(X^{T}X+\lambda \left[\begin{array}{ccccc}0& & & & \\ & 1& & & \\ & & 1& & \\ & & & \ddots & \\ & & & & 1\end{array}\right]{)}^{-1}{X}^{T}y$$

Non-invertibilty

If $m\le$ n, $m$ is the # of examples and $n$ is the # of features, then $X^{T}X$ is singular matrix.

But when you add the $\lambda$ part, this matrix is invertible.

Regularized logistic regression

Just like linear regression, we add the term to penalize $\theta$s in the cost function.

$J(\theta )=-[\frac{1}{m}{\sum }_{i=1}^m{y}^{(i)}\ast log(h_{\theta }(x^{(i)}))+(1-y^{(i)})log(1-h_{\theta }(x^{(i)}))]+\frac{\lambda }{m}{\sum }_{j=1}^n{\theta }_j^2$

Annotation

It has to be clear that although logistic and linear regression share the same math equation in the graident descent progress, they are two very different algorithm. They have their own hypothesis. The only reason they look alike in the gradient descent is that the cost fuction are different, so that we can unify the equation in graident descent.

Click here to see the following note.