Normal Equation

Deriving the Normal Equation

An alternate of Gradient Descent in Multiple Linear Regression

1. Problem Setup:

   1. Given a dataset with $m$ samples $(x^{(1)};x^{(2)};...;x^{(m)})$ with each sample having $n$ features (so $x_1^{(1)}$ denotes the value of the first feature of the first sample), and the response variables $(y^{(1)};...;y^{(m)})$ with $y^{(1)}$ being the “output” for the first sample, we want to calculate parameters ${\theta }_0,...,{\theta }_n$ such that the hypothesis $h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+...+{\theta }_m{x}_m$ gives the least-square linear equation for our dataset (i.e. for each pair of $(x_j^{(i)},y^{(i)})$ for all $i$ and $j$.
   2. For convenience of calculation, let $x_0^{(i)}=1$ for all $i$ so ${\vec{x}}^{(i)}=(x_0^{(i)},x_1^{(i)},x_2^{(i)},...,x_n^{(i)})$ is of the same dimension as $\vec{\theta }=({\theta }_0,...,{\theta }_n)$, so we rewrite the hypothesis as $h_{\theta }(x)={\theta }_0{x}_0+{\theta }_1{x}_1+...+{\theta }_m{x}_m$.

2. Derivation

   1. Cost Function: As “least square” means “minimizing the sum of squared loss”, the cost function is $J({\theta }_0,...,{\theta }_n)=$$\frac{1}{2m}$${\sum }_{i=1}^m({\theta }_0{x}_0^{(i)}+{\theta }_1{x}_1^{(i)}+...+{\theta }_m{x}_m^{(i)}-y^{(i)}{)}^2$ where $\frac{1}{2m}$ does not have any impact on the results. Our task is to minimize this function.

   2. Viewing $J$ as an inner product: In minimizing $J$, we are looking for $\theta$ values that can minimize the sum of a bunch of squared Euclidean distance.

      Notice that ${\sum }_{i=1}^m({\theta }_0{x}_0^{(i)}+{\theta }_1{x}_1^{(i)}+...+{\theta }_m{x}_m^{(i)}$