[Note] Machine Learning——Chapter 2

2-1 Model representation

Linear regression with one variable

2-2 Cost function

Also called squared error function.

• Hypothesis:

$$h_{\theta }(x)={\theta }_0+{\theta }_{1}x$$

• parameters: ${\theta }_0,{\theta }_1$

• Cost function:

$$J({\theta }_0,{\theta }_1)=\frac{1}{2m}\sum _{i=1}^m{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2$$

• Goal:

$$minimizeJ({\theta }_0,{\theta }_1)$$

2-3 Gradient descent

Have some function: $J({\theta }_0,{\theta }_1)$

Want: $minJ({\theta }_0,{\theta }_1)$

Outlines:

• Start with some ${\theta }_0,{\theta }_1$
• Keep changing ${\theta }_0,{\theta }_1$ to reduce $J({\theta }_0,{\theta }_1)$ until we hopefully end up at a minimum or maybe a local minimum

Gradient descent algorithm

repeat until convergence
$${\theta }_j:={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J({\theta }_0,{\theta }_1)(forj=0andj=1)$$

• “:=” means assignment, it is different from the truth assertion “=”.
• “α” means learning rate. And what alpha does is, it basically controls how big a step we take downhill with gradient descent. So if alpha is very large, then that corresponds to a very aggressive gradient descent procedure.

---

Correct: Simultaneous update

• $temp0:={\theta }_0-\alpha \frac{\partial }{\partial {\theta }_0}J({\theta }_0,{\theta }_1)$

• $temp1:={\theta }_1-\alpha \frac{\partial }{\partial {\theta }_1}J({\theta }_0,{\theta }_1)$

• ${\theta }_0:=temp0$

• ${\theta }_1:=temp1$

Incorrect:

• $temp0:={\theta }_0-\alpha \frac{\partial }{\partial {\theta }_0}J({\theta }_0,{\theta }_1)$
• ${\theta }_0:=temp0$
• $temp1:={\theta }_1-\alpha \frac{\partial }{\partial {\theta }_1}J({\theta }_0,{\theta }_1)$
• ${\theta }_1:=temp1$

---

• if α is too small, gradient descent can be slow.
• if α is too large, gradient descent can overshoot the minimum. It may fail to converge, or even diverge.

Gradient descent can converge to a local minimum, even with the learning rage α fixed.

As we approach a local minimum, gradient descent will automatically take smaller steps. So, no need to decrease α over time.

Gradient descent for linear regression

Gradient descent algorithm

repeat until convergence
$$\frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)=\frac{\partial }{\partial {\theta }_0}J({\theta }_0,{\theta }_1)$$

$$\frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x^{(i)}=\frac{\partial }{\partial {\theta }_1}J({\theta }_0,{\theta }_1)$$

$${\theta }_0:={\theta }_0-\alpha \frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)$$

$${\theta }_1:={\theta }_1-\alpha \frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)\cdot x^{(i)}$$

Does gradient descent on this type of cost function which you get whenever you’re using linear regression, it will always convert to the global optimum.

“Batch” Gradient Descent

“Batch”: Each step of gradient descent uses all the training examples. So, in gradient descent, when computing derivatives, we’re computing these sums, that sums over our M training examples.