机器学习Normal Equation的推导（不要求矩阵求导）

首先是代价函数$J(\theta )$的定义为
$$J(\theta )=\frac{1}{2m}\sum _{i=1}^m({\theta }_0{x}_0^{(i)}+{\theta }_1{x}_1^{(i)}+\cdots +{\theta }_n{x}_n^{(i)}-y^{(i)}{)}^2$$
对任意参数${\theta }_j$求偏导，得
$$\frac{\partial J(\theta )}{\partial {\theta }_j}=\frac{1}{m}\sum _{i=1}^m{x}_j^{(i)}({\theta }_0{x}_0^{(i)}+{\theta }_1{x}_1^{(i)}+\cdots +{\theta }_n{x}_n^{(i)}-y^{(i)})$$
要求极值，则偏导等于0，有
$$\frac{1}{m}\sum _{i=1}^m{x}_j^{(i)}({\theta }_0{x}_0^{(i)}+{\theta }_1{x}_1^{(i)}+\cdots +{\theta }_n{x}_n^{(i)}-y^{(i)})=0$$
尝试把求和号打开，化成矩阵形式
$$\left[\begin{array}{cccc}{x}_j^{(1)}& x_j^{(2)}& \cdots & x_j^{(m)}\end{array}\right]\left[\begin{array}{c}{\theta }_0{x}_0^{(1)}+{\theta }_1{x}_1^{(1)}+\cdots +{\theta }_n{x}_n^{(1)}-y^{(1)}\\ {\theta }_0{x}_0^{(2)}+{\theta }_1{x}_1^{(2)}+\cdots +{\theta }_n{x}_n^{(2)}-y^{(2)}\\ ⋮\\ {\theta }_0{x}_0^{(m)}+{\theta }_1{x}_1^{(m)}+\cdots +{\theta }_n{x}_n^{(m)}-y^{(m)}\end{array}\right]=0$$
上面的第二个矩阵还可以再次展开为
$$\left[\begin{array}{c}{\theta }_0{x}_0^{(1)}+{\theta }_1{x}_1^{(1)}+\cdots +{\theta }_n{x}_n^{(1)}-y^{(1)}\\ {\theta }_0{x}_0^{(2)}+{\theta }_1{x}_1^{(2)}+\cdots +{\theta }_n{x}_n^{(2)}-y^{(2)}\\ ⋮\\ {\theta }_0{x}_0^{(m)}+{\theta }_1{x}_1^{(m)}+\cdots +{\theta }_n{x}_n^{(m)}-y^{(m)}\end{array}\right]=\left[\begin{array}{cccc}{x}_0^{(1)}& x_1^{(1)}& \cdots & x_n^{(1)}\\ x_0^{(2)}& x_1^{(2)}& \cdots & x_n^{(2)}\\ ⋮& ⋮& \ddots & ⋮\\ x_0^{(m)}& x_1^{(m)}& \cdots & x_n^{(m)}\end{array}\right]\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ ⋮\\ {\theta }_n\end{array}\right]-\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right]$$
所以，对于任一参数${\theta }_j$，都有
$$\left[\begin{array}{cccc}{x}_j^{(1)}& x_j^{(2)}& \cdots & x_j^{(m)}\end{array}\right]\left(\left[\begin{array}{cccc}{x}_0^{(1)}& x_1^{(1)}& \cdots & x_n^{(1)}\\ x_0^{(2)}& x_1^{(2)}& \cdots & x_n^{(2)}\\ ⋮& ⋮& \ddots & ⋮\\ x_0^{(m)}& x_1^{(m)}& \cdots & x_n^{(m)}\end{array}\right]\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ ⋮\\ {\theta }_n\end{array}\right]-\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right]\right)=0$$
将所有参数$\theta$得到的方程全部联立，就可以得到
$$\left[\begin{array}{cccc}{x}_0^{(1)}& x_0^{(2)}& \cdots & x_0^{(m)}\\ x_1^{(1)}& x_1^{(2)}& \cdots & x_1^{(m)}\\ ⋮& ⋮& \ddots & ⋮\\ x_n^{(1)}& x_n^{(2)}& \cdots & x_n^{(m)}\end{array}\right]\left(\left[\begin{array}{cccc}{x}_0^{(1)}& x_1^{(1)}& \cdots & x_n^{(1)}\\ x_0^{(2)}& x_1^{(2)}& \cdots & x_n^{(2)}\\ ⋮& ⋮& \ddots & ⋮\\ x_0^{(m)}& x_1^{(m)}& \cdots & x_n^{(m)}\end{array}\right]\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ ⋮\\ {\theta }_n\end{array}\right]-\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right]\right)=0$$
令
$$X=\left[\begin{array}{cccc}{x}_0^{(1)}& x_1^{(1)}& \cdots & x_n^{(1)}\\ x_0^{(2)}& x_1^{(2)}& \cdots & x_n^{(2)}\\ ⋮& ⋮& \ddots & ⋮\\ x_0^{(m)}& x_1^{(m)}& \cdots & x_n^{(m)}\end{array}\right],\Theta =\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ ⋮\\ {\theta }_n\end{array}\right],Y=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right]$$
则上述方程可写为
$$X^T(X\Theta -Y)=0$$
展开，再化简
$$\begin{gathered} X^{T}X\Theta -X^{T}Y=0 \\ {X}^{T}X\Theta =X^{T}Y \\ \Theta =(X^{T}X{)}^{-1}{X}^{T}Y \end{gathered}$$
就可以得到Normal Equation
$$\Theta =(X^{T}X{)}^{-1}{X}^{T}Y$$

直接使用这个式子求参数的话，可以避免多次迭代，而且求解出来的结果更加严格。缺点是对矩阵$X^{T}X$求逆的时间复杂度是$O(n^3)$的，当参数较多（$n\ge 10^5$）时基本上就宣告破产了，应换用梯度下降。