Coursera Machine Learning Note - Week 1

Linear Regression with One Variable

Hypothesis function: $h_{\theta}(x)=\theta_{0}+\theta_{1}x$

Idea：Choose $\theta_{0},\theta_{1}$so that $h_{\theta}(x)$ is close to $y$ for our training examples$(x,y)$

$\underset{\theta_{0},\theta_{1}}{minimize} \frac{1}{2m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)})^{2}$

Parameters：$\theta_{0}, \theta_{1}$

Cost function：$J(\theta_{0},\theta_{1})=\frac{1}{2m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)})^{2}$, where $m$ is the training size

So, the goal: $\underset{\theta_{0},\theta_{1}}{minimize} J(\theta_{0},\theta_{1})$

Note that： $J(\theta_{0},\theta_{1})=\frac{1}{2m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)})^{2}=\frac{1}{2m}\sum_{i=1}^{m}((\theta_{0}+\theta_{1}x^{(i)})^{2}+(y^{(i)})^{2}-2(\theta_{0}+\theta_{1}x^{(i)})y^{(i)})$

It’s a function of the parameters $\theta_{0}, \theta_{1}$, its graph like the following:

Gradient descent algorithm：

1. Start with some $\theta_{0},\theta_{1}$
2. Keep changing $\theta_{0},\theta_{1}$to reduce $J(\theta_{0},\theta_{1})$ until we hopefully end up at a minimum

repeat until convergence{

$\theta_{0}:=\theta_{0}-\alpha\frac{\partial}{\partial\theta_{0}}J(\theta_{0},\theta_{1})=\theta_{0}-\alpha\frac{1}{m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)})$
$\theta_{1}:=\theta_{1}-\alpha\frac{\partial}{\partial\theta_{1}}J(\theta_{0},\theta_{1})=\theta_{1}-\alpha\frac{1}{m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)})\cdot x^{(i)}$
(update $\theta_{0},\theta_{1}$ simultaneously)
}
where $\alpha$ is learning rate, usually set to 0.03