Five Ways to Derive the Normal Equation
There are at least five different approaches to derive the normal equation. This post aims to provide a living document of the normal equation as well as its interpretations.
Notations
RSS
stands for Residual Sum Squared error,
β
denotes parameters in a form of column vector,
X
is an
Method 1. Vector Projection onto the Column Space
This is the most intuitive way to understand the normal equation. The optimization of linear regression is equivalent to finding the projection of vector
y
onto the column space of
As the projection is denoted by yˆ=Xβ , the optimal configuration of β is when the error vector y−Xβ is orthogonal to the column space of X , that is
Solving this gives:
Here XTX is denoted as the Gram Matrix and XTy is denoted as a moment matrix. More intuitively, the Gram matrix captures the correlations among the features and the moment matrix captures the contributions from each feature to the regression outcome.
Method 2. Direct Matrix Differentiation
This is the most straightforward way to solve the equation by rewriting S(β) into a simpler form:
Differentiate S(β) w.r.t. β :
Solving S(β) gives:
Method 3. Matrix Differentiation with Chain-rule
This is the simplest method for a lazy person, as it takes very little effort to reach the solution. The key is to apply the chain-rule:
solving S(β) gives:
This method requires an understanding of matrix differentiation of the quadratic form:
Method 4. Without Matrix Differentiation
We can rewrite S(β) as following:
where ⟨⋅,⋅⟩ is the inner product defined by
The idea is to rewrite S(β) into the form of S(β)=(x−a)2+b such that x can be solved exactly.
Method 5. Statistical Learning Theory
An alternative method to derive the normal equation arises from the statistical learning theory. The aim of this task is to minimize the expected prediction error given by:
where x stands for a column vector of random variables,
Differentiating EPE(β) w.r.t. β gives:
Before we proceed, let’s check the dimensions to make sure the partial derivative is correct. EPE is the expected error: a 1×1 vector. β is a column vector that is N×1 . According to the Jacobian in vector calculus, the resulting partial derivative should take the form
which is a 1×N vector. Looking back at the right-hand side of the equation above, we find 2(y−xTβ)(−1) being a constant while xT being a row vector, resuling the same 1×N dimension. Thus, we conclude the above partial derivative is correct. This derivative mirrors the relationship between the expected error and the way to adjust parameters so as to reduce the error. To understand why, imagine 2(y−xTβ)(−1) being the errors incurred by the current parameter configurations β and xT being the values of the input attributes. The resulting derivative equals to the error times the scales of each input attribute. Another way to make this point is: the contribution of error from each parameter βi has a monotonic relationship with the error 2(y−xTβ)(−1) as well as the scalar xT that was multiplied to each βi .
Now, let’s go back to the derivation. Because 2(y−xTβ)(−1) is 1×1 , we can rewrite it with its transpose:
∂EPE(β)∂β=∫2(y−xTβ)T(−1)xTPr(dx,dy) .
Solving ∂EPE(β)∂β=0 gives:
References
[1] Wikipedia: Linear Least Squares
[2] The elements of statistical Learning