Linear regression

m=Number of training examples
h(hypothesis)=output function

Linear regression with one variable. Univariate linear regression.

Idea: Choose $\theta_0,\theta_1$ so that $h_\theta(x)$is close to y for our training examples(x,y)

Cost Function:
squared error function

minimize J($\theta_0,\theta_1$)——overall objective for linear regression

Gradient Descent
outline:

start with some $\theta_0,\theta_1$(initialize them to 0)
(the result depending on where to start on the graph,however J($\theta_0,\theta_1$) is a convex function)

repeat until convergence{
$\theta_j:=\theta_j-\alpha\frac{\delta}{\delta\theta_j}J(\theta_0,\theta_1)\text{(for j=0 and j=1)}$
}
$\alpha\frac{\delta}{\delta\theta_j}J(\theta_0,\theta_1)$ is the derivative term
$\alpha$ is learning rate
*simultaneous update
$temp_0:=...$
$temp_1:=...$
$\theta_0=temp_0$
$\theta_1=temp_1$
if you do not do so,maybe it also works well, but we don’t call it GD algorithm.

compress:

if $\alpha$ is too small ,gradient descent can be slow.
if $\alpha$ is too large,gradient decent can overshoot the minimum.It may fail to converge, or even diverge.
GD can converge to a local minimum,even with the learning rate $\alpha$ fixed, because as we approach a local minimum, GD will automatically take smaller steps.

---

Linear Regression with Multiple Features
$X_j^{(i)}$ is the value of feature j in $i^{th}$ training example
$h_\theta(x)=\theta_0x_0+\theta_1x_1+...\theta_nx_n=\theta^TX$
(both with n+1 elements)
(define $x_0=1$)

---

feature scaling
if different features takes on similar ranges of values,FD will
converge more quickly.
It speeds up gradient descent by making it require fewer iterations to get to a good solution.
maybe
-3 to 3
-$\frac{1}{3}$ to $\frac{1}{3}$
is well
else $x_1$=$\frac{...}{10000}$
get every feature into approximately a $-1\le x_i \le 1$
range or standard deviation

mean normalization
makes features have approximately 0

to implement both measures above:
x = (value - average_value)/(max_value - min_value)

feature scaling doesn’t have to be too exact

converge judge

There is a example convergence test,but the threshold is not easy to find. So we had better plot the function.