Stanford ML - Lecture 1 - Linear regression with one variable

Model representation

Cost function

Cost function intuition I

Cost function intuition II

Gradient descent

• start with some \theta_0, \theta_1
• keep changing \theta_0, \theta_1 to reduce J(\theta_0, \theta_1) until we hopefully end up at a minimum

simultaneous update all parameters in the cost function, and \alpha is positive.

Gradient descent intuition

If \alpha is too small, gradient descent can be slow. If \alpha is too large, gradient descent can overshoot the minimum. It may fail to converge, or ever diverge.

As we approach a local minimum, gradient descent will automatically take smaller steps. So, no need to decrease \alpha over time.

Gradient descent for linear regression

“Batch” Gradient Descent: each step of gradient descent uses all the training samples.