CS229 Lecture Note(1): Linear Regression

1. LMS Algorithm

• The Ordinary Least Squares Regression Model:

  $$h(\theta)=\theta^Tx$$

• Cost Function:

  $$J(\theta)=\frac{1}{2}\sum_{i=1}^m(h_\theta(x^i)-y^i)^2$$

• Gradient Descent Algorithm:
  

  $$\theta:=\theta-\alpha \frac{\partial}{\partial\theta}J(\theta)$$

• LMS (least mean squares) update rule (also called for Widrow-Hoff learning rule):

  $$\theta_j:=\theta_j+\alpha\sum_{i=1}^m{(y^i-h_\theta(x^i))x_j^i}$$

• Batch Gradient Descent vs. Stochastic Gradient Descent

  
  # BGD
  
  Repeat until convergence {
      theta = theta + alpha * sum((y_i - h_i) * x_i)
  }
  
  
  # SGD
  
  Loop {
      for i=1 to m, {
          theta = theta + alpha * (y_i - h_i) * x_i
      }
  }

• Normal Equation Solution:

  $$\theta=(X^TX)^{-1}X^TY$$

2. Probabilistic Interpretation

• Predictive Probability Assumption: a Gaussian Distribution

  $$p(y|x;\theta)=\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(y-\theta^Tx)^2}{2\sigma^2}}\sim\mathcal{N}(\theta^Tx,\sigma^2)$$

• Likelihood Function of $\theta$: the probability of the given data $y$ (given i.i.d. assumption)

  $$L(\theta)=\prod_{i=1}^m{p(y^i|x^i;\theta)}=\prod_{i=1}^m{\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(y^i-\theta^Tx^i)^2}{2\sigma^2}}}$$

• Maximum Likelihood Method: choose $\theta$ to maximize $L(\theta)$ or the log likelihood $l(\theta)$:

  $$l(\theta)=\log{L(\theta)}\Rightarrow\frac{1}{2}\sum_{i=1}^m{(y^i-\theta^Tx^i)^2}$$ $$\theta=\arg\max_\theta{l(\theta)}$$

The least-squares regression model corresponds to the maximum likelihood estimation of $\theta$ under a Gaussian distribution assumption on data.

3. Locally Weighted Linear Regression

• Motivation: get rid of the problem of feature selection (which leads to the underfitting and overfitting problems)

• Parametric vs. Non-parametric learning algorithm

• LWR algorithm:
  Querying a certain point $x$,

  1. Fit $\theta$ to minimize $\sum_i{w^i(y^i-\theta^Tx^i)^2}$, where $w^i=e^{-\frac{(x^i-x)^2}{2\tau^2}}$

  2. Output $\theta^Tx$

  3. Hence, the (errors on) training examples close to the query point $x$ would be given a much higher weight to determine $\theta$ (local linearity).