【公式推导】正规方程

---

线性代数基础：

假设向量$A=\left[\begin{array}{cccc}a& b& c& d\end{array}\right]$，则$A^T=\left[\begin{array}{c}a\\ b\\ c\\ d\end{array}\right]$

$$A^2=A\ast A^T=\left[\begin{array}{cccc}a& b& c& d\end{array}\right]\ast \left[\begin{array}{c}a\\ b\\ c\\ d\end{array}\right]$$

---

矩阵求导基础：

$\frac{\partial x^T}{\partial x}=1$	$\frac{\partial x}{\partial x^T}=1$
$\frac{\partial Ax}{\partial x}=A^T$	$\frac{\partial xA}{\partial x}=A^T$
$\frac{\partial Ax}{\partial x^T}=A$	$\frac{\partial x^{T}A}{\partial x}=A$

---

损失函数:

$$\begin{gathered} J(\theta )=(h(x_1)-y_1{)}^2+(h(x_2)-y_2{)}^2+...+(h(x_m)-y_m{)}^2 \\ =\sum _{i=1}^m(h(x_i)-y_i{)}^2 \\ =(h(x)-y{)}^T\ast (h(x)-y) \end{gathered}$$

---

正规方程推导

• 把该损失函数转换成矩阵写法：

$$J(\theta )=(X\theta -y{)}^T\ast (X\theta -y)$$

其中y是真实值矩阵，X是特征值矩阵，$\theta$是权重矩阵

• 为了取到代价函数对最小值，所以对损失函数进行求导并使其等于0

$$\frac{\partial (J(\theta ))}{\partial (\theta )}=\frac{\partial [(X^T{\theta }^T-y^T)\ast (X\theta -y)]}{\partial (\theta )}$$

$$=\frac{\partial (X^T{\theta }^{T}X\theta -X^T{\theta }^{T}y-y^{T}X\theta +y{y}^T)}{\partial (\theta )}$$

$$=X^{T}X\theta +X\theta X^T-X^{T}y-y{X}^T=0$$

故：
$$X^{T}X\theta =X^{T}y$$

$$(X^{T}X)\ast (X^{T}X{)}^{-1}\theta =X^{T}y\ast (X^{T}X{)}^{-1}$$

$$\theta =(X^{T}X{)}^{-1}{X}^{T}y$$

由此推导出正规方程