吴恩达机器学习 Chapter5 多元线性回归（Linear Regression with multiple variables）

Multiple variables => Multiple features

hypothesis

Vectorize

$h_{\theta }(x)={\theta }^{T}x$
$x\in R^{n+1},\theta \in R^{n+1}$

Cost function

Cost function remains the same. The parameters are vectorized.

New algorithm

NOTE: Still, the parameters should be updated simultaneously.

Gradient Descent Practice

Feature Scaling

Idea

Get every feature into approximately a $-1\le x\le 1$ range. To make sure that the function will converge quickly.

Mean normalization

$x_i=>\frac{x-{\mu }_i}{s_i}$
${\mu }_i$: the mean of the feature $i$
$s_i$ : the range of the feature $i$ ($max-min$) or standard deviation

NOTE: Do not apply to $x_0=1$!

Learning Rate

Debugging

To make sure gradient descent is working correctly.
Ploting $J(\theta )$:

• To debugging: $J(\theta )$ should decrease after every iteration.
  If gradient descent not work like this:
  
  Try to use smaller $\alpha$
• To judge whether convergengce: Declare convergence when $J(\theta )$ decrease by less than $10^{-3}$ in one iteration.

Summary: Not too big nor too small

• If too small: Slow to converge
• If too big: $J(\theta )$ may not decrease every iteration; may not converge.
• To try $\alpha$: try …0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, …

Another solution: Normal equation

A method to solve for $\theta$ analytically.
Not work for complex algorithm.

Intuition

To get the minimum, set $\frac{\partial J(\theta )}{\partial {\theta }_j}=0$ for every $j$
$\theta =(X^{T}X{)}^{-1}{X}^{T}y$
Feature scaling is unnecessary.

Advantage and Disadvantage (Compared to GD)

m training examples, n features

Gradient Descent	Normal Equation
Need to choose $\alpha$	No need to choose $\alpha$
Needs many iterations	One time. No iteration.
Works well even when n is large	Need to compute $(X^{T}X{)}^{-1}$ $O(n^3)$ slow if n is large
If $n>10,000$	If $n<10,000$

Normal equation and non-invertibility

When $X^{T}X$ is non-invertible?

• Redundant features(linearly dependent)
• Too many features($m\le n$)
  => Delete some features; or use regularization.