Normal Equation

Deriving the Normal Equation

An alternate of Gradient Descent in Multiple Linear Regression

1. Problem Setup:

   1. Given a dataset with $m$ samples $(x^{(1)};x^{(2)};...;x^{(m)})$ with each sample having $n$ features (so $x_1^{(1)}$ denotes the value of the first feature of the first sample), and the response variables $(y^{(1)};...;y^{(m)})$ with $y^{(1)}$ being the “output” for the first sample, we want to calculate parameters ${\theta }_0,...,{\theta }_n$ such that the hypothesis $h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+...+{\theta }_m{x}_m$ gives the least-square linear equation for our dataset (i.e. for each pair of $(x_j^{(i)},y^{(i)})$ for all $i$ and $j$.
   2. For convenience of calculation, let $x_0^{(i)}=1$ for all $i$ so ${\vec{x}}^{(i)}=(x_0^{(i)},x_1^{(i)},x_2^{(i)},...,x_n^{(i)})$ is of the same dimension as $\vec{\theta }=({\theta }_0,...,{\theta }_n)$, so we rewrite the hypothesis as $h_{\theta }(x)={\theta }_0{x}_0+{\theta }_1{x}_1+...+{\theta }_m{x}_m$.

2. Derivation

   1. Cost Function: As “least square” means “minimizing the sum of squared loss”, the cost function is $J({\theta }_0,...,{\theta }_n)=$$\frac{1}{2m}$${\sum }_{i=1}^m({\theta }_0{x}_0^{(i)}+{\theta }_1{x}_1^{(i)}+...+{\theta }_m{x}_m^{(i)}-y^{(i)}{)}^2$ where $\frac{1}{2m}$ does not have any impact on the results. Our task is to minimize this function.

   2. Viewing $J$ as an inner product: In minimizing $J$, we are looking for $\theta$ values that can minimize the sum of a bunch of squared Euclidean distance.

      Notice that ${\sum }_{i=1}^m({\theta }_0{x}_0^{(i)}+{\theta }_1{x}_1^{(i)}+...+{\theta }_m{x}_m^{(i)}-y^{(i)}{)}^2$ is the same as the following inner product, which returns a scalar output:

      ​ ${\left[\begin{array}{c}{\theta }_0{x}_0^{(1)}+...+{\theta }_n{x}_n^{(1)}-y^{(1)}\\ {\theta }_0{x}_0^{(2)}+...+{\theta }_n{x}_n^{(2)}-y^{(2)}\\ ........\\ {\theta }_0{x}_0^{(m)}+...+{\theta }_n{x}_n^{(m)}-y^{(m)}\end{array}\right]}^{\mathbf{T}}$$\left[\begin{array}{c}{\theta }_0{x}_0^{(1)}+...+{\theta }_n{x}_n^{(1)}-y^{(1)}\\ {\theta }_0{x}_0^{(2)}+...+{\theta }_n{x}_n^{(2)}-y^{(2)}\\ ........\\ {\theta }_0{x}_0^{(m)}+...+{\theta }_n{x}_n^{(m)}-y^{(m)}\end{array}\right]$

      ​ (You may expand this to check how it equals the algebraic expression above)

   3. Viewing the vector used above as a matrix-vertor product subtracting another vectior :

      $\bullet$ The matrix $x\left[\begin{array}{cccc}{x}_0^{(1)}& .& .& x_n^{(1)}\\ x_0^{(2)}& .& .& x_n^{(2)}\\ .& .& .& .\\ x_0^{(m)}& .& .& x_n^{(m)}\end{array}\right]$timing the vector $\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ .\\ .\\ {\theta }_n\end{array}\right]$ gives the vector $\left[\begin{array}{c}{\theta }_0{x}_0^{(1)}+...+{\theta }_n{x}_n^{(1)}\\ {\theta }_0{x}_0^{(2)}+...+{\theta }_n{x}_n^{(2)}\\ ........\\ {\theta }_0{x}_0^{(m)}+...+{\theta }_n{x}_n^{(m)}\end{array}\right]$ .

      $\bullet$ We denote $\left[\begin{array}{cccc}{x}_0^{(1)}& .& .& .x_n^{(1)}\\ x_0^{(2)}& .& .& .x_n^{(2)}\\ .& .& .& .\\ x_0^{(m)}& .& .& .x_n^{(m)}\end{array}\right]$, the matrix, as $X$, and the vector, $\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ .\\ .\\ {\theta }_n\end{array}\right]$, as $\vec{\theta }$.

      $\bullet$ Therefore, $\left[\begin{array}{c}{\theta }_0{x}_0^{(1)}+...+{\theta }_n{x}_n^{(1)}-y^{(1)}\\ {\theta }_0{x}_0^{(2)}+...+{\theta }_n{x}_n^{(2)}-y^{(2)}\\ ........\\ {\theta }_0{x}_0^{(m)}+...+{\theta }_n{x}_n^{(m)}-y^{(m)}\end{array}\right]$ is $X\vec{\theta }-\vec{y}$. The inner product is $(X\vec{\theta }-\vec{y}{)}^{\mathbf{T}}(X\vec{\theta }-\vec{y})$.

   4. We minimize $(X\vec{\theta }-\vec{y}{)}^{\mathbf{T}}(X\vec{\theta }-\vec{y})$. Let $(X\vec{\theta }-\vec{y}{)}^T(X\vec{\theta }-\vec{y})$ be a function $E$ of the unknown vector $\vec{\theta }$. If the system $X\vec{\theta }-\vec{y}=\vec{0}$ has a unique solution, then that’s enought as $\min E\ge 0$. However, when this system fails to be solvable, we have to minimize $E$ using some techniques. The following contents show two ways of minimization

   5. Two ways to solve for $\vec{\theta }$:

      1. Taking the partial derivative of $E$ with respect to the unknown vector $\vec{\theta }$:

         $\bullet$ $\frac{\partial E}{\partial \vec{\theta }}$$=2X^{\mathbf{T}}(X\vec{\theta }-\vec{y})$ by the product rule.

         $\bullet$ To get the local extrema, we let $X^{\mathbf{T}}(X\vec{\theta }-\vec{y})=\vec{0}$. $X^{\mathbf{T}}X\vec{\theta }-X^{\mathbf{T}}\vec{y}=\vec{0}$, so $\vec{\theta }=(X^{\mathbf{T}}X{)}^{-1}{X}^{\mathbf{T}}\vec{y}$ after symplifying. However, this solution only applies to the situation where $X^{\mathbf{T}}X$ is invertible.

         $\bullet$ Therefore, $\min J(\vec{\theta })=$$\frac{1}{2m}$${\sum }_{i=1}^m({\overrightarrow{{\theta }^{\ast }}}^{\mathbf{T}}{\vec{x}}^{(i)}-y^{(i)}{)}^2$ =$\frac{1}{2m}$${\sum }_{i=1}^m([X^{\mathbf{T}}X{)}^{-1}{X}^{\mathbf{T}}\vec{y}{]}^{\mathbf{T}}{\vec{x}}^{(i)}-y^{(i)}{)}^2$ (notice that we write $J$ as a function of the solved vector ${\vec{\theta }}^{\ast }$), and our hypothesis is $h_{\theta }(\vec{x})={\vec{\theta }}^{\mathbf{T}}\vec{x}$ (or $\overrightarrow{x^{\mathbf{T}}}\vec{\theta }$) where $\vec{x}$ is some arbitrary features sealed in a vector with its first entry being $1$.

         $\bullet$ When $X$ has more columns than rows (i.e. in the dataset, we have more features than samples; a row in $X$ represents one sample, and a column represents one unknown), the system $X\vec{\theta }-\vec{y}$ typically has infinitely many solutions, which means any one of them can be used for $\vec{\theta }$ depending on the preference of the model.

      2. Orthogonally projecting $\vec{y}$ onto $X$'s column space:

         $\bullet$ Minimizing $E$ means to minimize the length of the vector $(X\vec{\theta }-\vec{y})$. Intuitively, this lead to the idea of choosing $\vec{\theta }$ so that $X\vec{\theta }$ is the result of orthogonally projecting $\vec{y}$ onto the image of $X$ (or the column space, or span of the columns of $X$).

         $\bullet$ The formula for orthorgonal projection onto the image of $X$ is $X(X^{\mathbf{T}}X{)}^{-1}{X}^{\mathbf{T}}$, so if $X\vec{\theta }=X(X^{\mathbf{T}}X{)}^{-1}{X}^{\mathbf{T}}\vec{y}$, $\vec{\theta }=(X^{\mathbf{T}}X{)}^{-1}{X}^{\mathbf{T}}\vec{y}$.