吴恩达·Machine Learning || chap4 Linear Regression with multiple variables 简记

4 Linear Regression with multiple variables

4-1 Multiple features

Multiple features (variables)

​ Notation:

​ $n$​ = number of features
​ $x^{(i)}$​=input(features) of $i^{th}$​ training example.
​ $x_j^{(i)}$​=value of feature in $i^{th}$​​ training example.

Hypothesis

​ $h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+\cdot \cdot \cdot +{\theta }_n{x}_n$

​ define $x_0$=1

​ $x=\left[\begin{array}{c}{x}_0\\ x_1\\ x_2\\ ⋮\\ x_n\end{array}\right]$ $\theta =\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ {\theta }_2\\ ⋮\\ {\theta }_n\end{array}\right]$

​ $h_{\theta }(x)={\theta }^{T}x$

4-2 Gradient descent for multiple variable

Cost function
$$J(\theta )=J({\theta }_0,{\theta }_1,\cdots ,{\theta }_n)=\frac{1}{2m}\sum _{i=1}^m(h_{\theta }(x^{(i)}-y^{(i)}{)}^2$$
Gradient descent

​ ${\theta }_j:={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J(\theta )$​​

​ Previously(n=1):

​ Repeat{
$$\{\begin{array}{l}{\theta }_0:={\theta }_0-\alpha \frac{1}{m}{\sum }_{i=1}^m(h_{\theta }(x^{(i)}-y^{(i)})\\ {\theta }_1:={\theta }_1-\alpha \frac{1}{m}{\sum }_{i=1}^m(h_{\theta }(x^{(i)}-y^{(i)}).x^{(i)}\end{array}$$
​ }

​ New algorithm(n$\ge 1$​)

​ Repeat{
$${\theta }_j:={\theta }_j-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)}-y^{(i)})x_j^i$$
​ }

4-3 Gradient descent in practice I: Feature Scaling

Feature Scaling 特征缩放

​ Ideal ：make sure features are on a similar scale

​ Get every feature into approximately a $0\le x_i\le 1$​ ($x_0=1$) range​​

Mean normalization

​ Replace $x_i$ with $x_i-{\mu }_i$ to make features have approximately zero mean (Do not apply to $x_0=1$)
$$x_i⟵\frac{x_i-{\mu }_i}{s_i}$$
​ $x_i$: feature number

​ ${\mu }_i$:average value

​ $s_i$: range(max-min)

4-4 Gradient descent in practice I: Learning rate

Gradient descent

​ ${\theta }_j:={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J(\theta )$

• Debugging: make sure gradient descent is working correctly

• How to choose learning rate $\alpha$

  Convergence test:

  ​ Declare convergence if $J(\theta )$​​ decreases by less than $10^{-3}$​​​ ($ϵ$​​)in one iteration

Summary

For sufficiently small $\alpha$, $J(\theta )$​should decrease on every iteration.
But if Q is too small, gradient descent can be slow to converge

To choose $\alpha$,try

​ steps of ten

​ ···,0.001,0.003,0.01,0.03,···

4-5 Features and polynomial regression

housing price prediction

Polynomial regression

$h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_3$

$⟶$$h_{\theta }(x)={\theta }_0+{\theta }_1(size)+{\theta }_2(size{)}^2+{\theta }_3(size{)}^3$​

Choice of features

$h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2\sqrt{size}$

4-6 Normal equation

Intuition: ($\theta \in R$​)($\theta \in R^{n+1}$)
$$\theta =(X^{T}X{)}^{-1}{X}^{T}y$$

pinv(x'*x)*x'*y		%octave

m examples ($(x^{(1)},y^{(1)}),\cdots ,(x^{(m)},y^{(m)})$​​),n features

$x^{(i)}=\left[\begin{array}{c}{x}_0^{(i)}\\ x_1^{(i)}\\ x_2^{(i)}\\ ⋮\\ x_n^{(i)}\end{array}\right]$​​ X=$\left[\begin{array}{ccc}\cdots & （x^{(1)}{)}^T& \cdots \\ \cdots & （x^{(2)}{)}^T& \cdots \\ & ⋮& \\ \cdots & （x^{(m)}{)}^T& \cdots \end{array}\right]$​​

$y=\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ ⋮\\ y^{(m)}\end{array}\right]$

Gradient Descent O(n)	Normal Equation O($n^3$)
Need to choose $\alpha$	No need to choose $\alpha$
Needs many iterations	Don’t need to iterate
Works well even when n is large	Need to compute $(X^{T}X{)}^{-1}$
	Slow if n is very large

4-7 Normal equation and non-invertibility(optional)

singular or degenerate matrices

What if $X^{T}X$ is non-invertible ?

• Redundant features (linear dependent)

  e.g. $x_1$=size in $fee{t}^2$​

  ​ $x_2$=size in $m^2$​

• Too many features(e.g. m$\le$n)

  -Delete some features,or use regulariztion

**Octave Tutorial

Working on and submitting programming exercises