Machine Learning Lecture Note 002 -- Multivariate Linear Regression

Multivariate Linear Regression

Notation

$x_j^{(i)}$ = value of feature j in the $i^{th}$ training example

$x^{(i)}$ = the input (features) of the $i^{th}$ training example

Hypothesis Function

$h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+...+{\theta }_n{x}_n$

Rewrite in matrix notation:

$h_{\theta }(x)=[{\theta }_0{\theta }_1{\theta }_2...{\theta }_n]\left[\begin{array}{c}{x}_0\\ x_1\\ x_2\\ ...\\ x_n\end{array}\right]={\theta }^{T}x$, where $x_0=1.$

Gradient Descent

For each feature we have:

Feature Scaling

Scale feature values within range [-1, 1]

i.e. If $x_i\in [0,2000]$, scale it by dividing 2000, then we have $x_i\in [0,1]$

The denominator is known as the scaling factor

Scaling factor $S=Max(x)-Min(x)$,
i.e. S 一般取 feature $x$ 的range

Mean Normalization

Replace $x_i$ with $\frac{x_i-\mu }{\sigma }$ （divide by SD）

OR with $\frac{x_i-\mu }{S}$ (divide by scaling factor, or the range)

Feature Scaling & Mean Normalization 有助于 Gradient Descent 更快 Converge

Choosing the right learning Rate ($\alpha$)

Good $\alpha$ values to try: 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, …

• Method 1: Plot Cost Function over each iteration
  
  （浅蓝converge)

如果plot不是decreasing function, reduce $\alpha ,$ or it won’t converge.

图A：$\alpha$ is good
图B：$\alpha$ is too small, slow convergence
图C：$\alpha$ is too big, diverge

• Method 2: Use convergence detection algorithm

  如果Cost Function每次下降的值低于某个threshold, 报converge

Threshold 一般取 small value such as $10^{-3}$