Multivariate Linear Regression

Multivariate Linear Regression

Multiple Features

Multivariate linear regression: Linear regression with mutiple variables.

Notation

$n$= number of features

$x^{(i)}$= input (features) of $i^{th}$ training example.

$x_j^i$= value of feature $j$ in $i^{th}$ training example.

$h_{\theta }(x)$=${\theta }_0+{\theta }_1{x}_1+...{\theta }_n{x}_n$

For convenience of notation, define $x_0$=1.(That is,$x_0^{(i)}$=1)
$$x=\left[\begin{array}{c}{x}_0\\ x_1\\ \cdots \\ x_n\end{array}\right]\in {\mathbb{R}}^{n+1}$$

$$\theta =\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ \cdots \\ {\theta }_n\end{array}\right]\in {\mathbb{R}}^{n+1}$$

$h_{\theta }(x)$=${\theta }^{T}x$

Gradient Descent for Multiple Variable

The gradient descent equation is generally the same form, we just have to repeat it for ‘n’ features:

Repeat until convergence:{

${\theta }_j$:=${\theta }_j$-$\alpha \frac{1}{m}{\sum }_{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})\cdot x_j^{(i)}$

}

Gradient Descent in Practice I- Feature Scaling

Idea: Make sure features are on similar scale.

Modify the ranges of our input variables: Speed up gradient descent by having each of input values in roughly the same range.(make the contour of cost function $J$ can become less skewed)

Because $\theta$ will descend quickly on small ranges and slowly on large ranges. This will oscillate inefficiently down to the optimum when the variables are very uneven.

No exact requirements.

Two techniques:

• feature scaling :Dividing the input values by the range of the input variables

  Get every feature into approximately a $-1\le x_i\le 1$ range.

• mean normalization: Subtracting the average value for an input variable from the values for that input variable.

  Replace $x_i$ with $x_i-{\mu }_i$ to make features have approximately zero mean (Do not apply to $x_0=1$)

Adjust input values in the formula:

$x_i:=\frac{x_i-{\mu }_i}{s_i}$

$\mu$ is the average of all the values for features(i) and $s_i$ is the range of values (max-min), or $s_i$ is the standard deviation.

• Dividing by the range, or dividing by the standard deviation, give different results.

Gradient Descent in Practice II- Learning Rate

• Debugging gradient descent

Make a plot with number of iterations on the x-axis. Now plot the cost function, J(θ) over the number of iterations of gradient descent. If J($\theta$) ever increases, then we need to increase $\alpha$.

($J_{\theta }$ should decrease after every iteration)

• Automatic convergence test

Declare convergence if J(θ) decreases by less than E in one iteration, where E is some small value such as $10^{-3}$.

If learning rate α is sufficiently small, then J(θ) will decrease on every iteration.

If α is too small: slow convergence.

If α is too large: may not decrease on every iteration and thus may not converge.

Features and Polynomial Regression

Improve features and the form of hypothesis

• combine multiple features into one

Polynomial Regression

• change the behavior or curve

(making it a quadratic, cubic or square root function (or any other form))

Keep in mind that, if you choose your features this way then feature scaling becomes very important.