Parameter Learning

Parameter Learning

Gradient Descent

1. The outline of gradient descent:

   • Start with some ${\theta }_0,{\theta }_1$ (commonly set ${\theta }_0=0,{\theta }_1=0$)

   • Keep changing ${\theta }_0,{\theta }_1$ to reduce $J({\theta }_0,{\theta }_1)$ until we hopefully end up at a minimum or a local minimum.

     Direction: derivative of cost function.(The slope of tangent is the derivative at the point )

Make steps down the cost function in the direction with the steepest descent

2. Property :

   The first time we ran the gradient descent, started at a slightly different location would’ve end up with a very different local optimum.

3. Assignment and Assertion(:= and =)

4. The gradient descent algorithm is:
   

   1） $\alpha$ (learning rate) controls how big a step we take

   2）Simultaneously update ${\theta }_0...{\theta }_n$

Gradient Descent Intuition

1. The $\alpha$ should not be too small ,or else it would the gradient descent can be slow. It also should not be too large, or else the gradient descent can overshoot the minimum. It may fail to converge or even to be diverge.

2. Gradient descent can converge to local minimum with fixed learning rate $\alpha$.

   Reason: As we approach a local minimum, gradient descent will automatically take smaller steps, so it is no need to decrease $\alpha$ over time.

Gradient Descent For Linear Regression

1. Derive a new form of the gradient descent equation
   

m is the size of the training set, ${\theta }_0$ is a constant that will be changing simultaneously with ${\theta }_1$ and $x_i$, $y_i$ are values of the given training set (data).

2. A derivation of $\frac{\partial }{\partial {\theta }_j}J(\theta )$ for a single example :

3. Convex function: bow-shaped function

   Has no other local optimum except for the one global optimum

4. "Batch" Gradient Descent: Every step of gradient descent uses all of the training example.