[ML of Andrew Ng]Week 1 : Linear Regression with One Variable

Week 1 Linear Regression with One Variable

Introduction

Two definitions of ML

• The field of study that gives computers the ability to learn without being explicitly programmed.
• A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.

Two types of ML

• Supervised Learning and Unsupervised Learning.
• The keys of supervised learning are the correct output and labeled examples.
• While data in Unsupervised learning has no labels,they are same.We don’t know what to do.

Prerequisite for this course

• Knowledge of basic computer science principles and skills, at a level sufficient to write a reasonably non-trivial computer program.
• Familiarity with the basic probability theory. (CS109 or Stat116 is sufficient but not necessary.)
• Familiarity with the basic linear algebra (any one of Math 51, Math 103, Math 113, or CS 205 would be much more than necessary.)

---

Linear Regression with One Variable

The Hypothesis Function

$$h_\theta (x) = \theta_0 + \theta_1 x$$
or:
$$h_\theta (x) = \theta_0 x_0 + \theta_1 x_1 \quad which \; x_0 = 1$$
PS: $x_j^{(i)}$ means the $j$rd features and $i$rd examples.

we can get the vectors $\boldsymbol{\theta}$ and $\boldsymbol{X}$ as:

$$\boldsymbol{\theta} = \begin{bmatrix} \theta_0 \\ \theta_1 \end{bmatrix}$$ $$\qquad (2 \times 1)$$
and:
$$\boldsymbol{X} = \begin{bmatrix} 1 & x^{(1)} \\ 1 & x^{(2)} \\ \vdots & \vdots \\ 1 & x^{(m)} \end{bmatrix}$$ $$\qquad (m \times 2)$$
So we get the $$\boldsymbol{H} = \boldsymbol{X}\boldsymbol{\theta}$$ $$(m \times 2) \times (2 \times 1) = (m \times 1)$$ like as:
$$\boldsymbol{H} = \begin{bmatrix} h_{\theta}(x^{(1)}) \\ h_{\theta}(x^{(2)}) \\ \vdots \\h_{\theta}(x^{(m)}) \end{bmatrix}$$ $$\qquad (m \times 1)$$

In matlab:

h = X*theta;

Cost Function

$$J(\theta_0 , \theta_1) = \frac{1}{2m} \sum_{i=1}^m (h_ \theta (x^{(i)}) - y^{(i)})^2$$

Attention: $J(\theta_0 , \theta_1)$　is a scalar, just a number.
In matlab, we can use like:

J = 1/(2*m) * sum((X*theta - y).^2);
%.^ means dot product
%sum means sum all elements in matrix

Gradient Descent

$$\theta_j = \theta_j - \alpha \frac{\partial}{\partial \theta_j} J(\theta_0 , \theta_1)$$
repeat until convergence.
parameter $\alpha$: Learning rate
parameter $\partial$: Slope of tangent aka derivative

Gradient Descent for Linear Regression

When specifically applied to the case of linear regression, a new form of the gradient descent equation can be derived.

$$\theta_j = \theta_j - \alpha \frac{1}{m} \sum_{i=1}^m ((h_ \theta (x_j^{(i)}) - y^{(i)}) x_j^{(i)})$$
In matlab, we can use like:

theta = theta - (alpha/m * X' * (X*theta - y));
%because of the matrix multiplication, we  need not sum them.