MIT线性代数笔记-第16讲-投影矩阵，最小二乘

16.投影矩阵，最小二乘

1. 证明两个极端情况

   证明当$\vec{b}$在$A$的列空间里时，它的投影为它本身：

   ​ 设$\vec{b}=A\vec{x}$，则$P\vec{b}=A(A^{T}A{)}^{-1}{A}^{T}A\vec{x}=A\vec{x}=\vec{b}$

   证明当$\vec{b}$与$A$的列空间正交时，它的投影为$\vec{0}$：

   ​ 因为$\vec{b}$与$A$的每一列都正交，所以$A^T\vec{b}=\vec{0}$，所以$P\vec{b}=A(A^{T}A{)}^{-1}{A}^T\vec{b}=\vec{0}$

2. 当使用$P$将$\vec{b}$投影到某个空间中得到$\vec{p}$时，$\vec{b}$分解出的另一个向量$\vec{e}$即为$\vec{b}$在该空间的任意正交补中的投影，此时那个正交补的投影矩阵即为$I-P$，由此可以得到两个互为正交补的空间的投影矩阵之和为$I$

3. 最小二乘法

   点为$(x_1,y_1),\cdots ,(x_n,y_n)$，设拟合直线为$\hat{y}=\hat{b}x+\hat{a}$

   有$A=\left[\begin{array}{cc}{x}_1& 1\\ ⋮& ⋮\\ x_n& 1\end{array}\right],\hat{\vec{x}}=\left[\begin{array}{c}\hat{b}\\ \hat{a}\end{array}\right],\vec{b}=\left[\begin{array}{c}{y}_1\\ ⋮\\ y_n\end{array}\right]$

   要使总误差最小，需要在$A$的列空间中找到一个${\vec{b}}^{{}^{\prime }}$使得$|\vec{b}-{\vec{b}}^{{}^{\prime }}|$最小（也就是使$(y_1-\widehat{y_1}{)}^2+\cdots +(y_n-\widehat{y_n}{)}^2$最小）

   证明${\vec{b}}^{{}^{\prime }}=\vec{p}$时误差最小：

   ​    此时误差为$|\vec{e}|$

   ​    若选取除$\vec{p}$以外的$A$的列空间中的向量，设其为${\vec{b}}^{{}^{\prime }}=\vec{p}+\vec{a}$，那么$\vec{b}-{\vec{b}}^{{}^{\prime }}=\vec{e}-\vec{a}$

   ​    已知$\vec{e}$垂直于$A$的列空间中的任意向量，又$\vec{a}$属于$A$的列空间，所以$\vec{e}$与$\vec{a}$垂直

   ​    因而$|\vec{e}-\vec{a}{|}^2={\vec{e}}^2+{\vec{a}}^2>{\vec{e}}^2$，即$|\vec{e}-\vec{a}|>|\vec{e}|$

   ​    所以选取除$\vec{p}$以外的$A$的列空间中的向量都会导致误差增大

   有$\vec{p}=(\widehat{y_1},\cdots ,\widehat{y_n}),\vec{e}=(y_1-\widehat{y_1},\cdots ,y_n-\widehat{y_n})$

   推导最小二乘法的公式：

   ​    法一：$\because P=A(A^{T}A{)}^{-1}{A}^T$

   ​       $\begin{array}{rl}\therefore \vec{p}& =\left[\begin{array}{cc}{x}_1& 1\\ ⋮& ⋮\\ x_n& 1\end{array}\right](\left[\begin{array}{ccc}{x}_1& \cdots & x_n\\ 1& \cdots & 1\end{array}\right]\left[\begin{array}{cc}{x}_1& 1\\ ⋮& ⋮\\ x_n& 1\end{array}\right]{)}^{-1}\left[\begin{array}{ccc}{x}_1& \cdots & x_n\\ 1& \cdots & 1\end{array}\right]\left[\begin{array}{c}{y}_1\\ ⋮\\ y_n\end{array}\right]\\ & =\left[\begin{array}{cc}{x}_1& 1\\ ⋮& ⋮\\ x_n& 1\end{array}\right]{\left[\begin{array}{cc}\sum x_i^2& n\overline{x}\\ n\overline{x}& n\end{array}\right]}^{-1}\left[\begin{array}{c}\sum x_i{y}_i\\ n\overline{y}\end{array}\right]\\ & =\left[\begin{array}{cc}{x}_1& 1\\ ⋮& ⋮\\ x_n& 1\end{array}\right]\left[\begin{array}{cc}\frac{1}{t}& \frac{-n\overline{x}}{nt}\\ \frac{-\overline{x}}{t}& \frac{\sum x_i^2}{nt}\end{array}\right]\left[\begin{array}{c}\sum x_i{y}_i\\ n\overline{y}\end{array}\right]\\ & =\left[\begin{array}{cc}\frac{n(x_1-\overline{x})}{nt}& \frac{\sum x_i^2-n\overline{x}{x}_1}{nt}\\ ⋮& ⋮\\ \frac{n(x_n-\overline{x})}{nt}& \frac{\sum x_i^2-n\overline{x}{x}_n}{nt}\end{array}\right]\left[\begin{array}{c}\sum x_i{y}_i\\ n\overline{y}\end{array}\right]\\ & =\left[\begin{array}{c}\frac{1}{t}(\sum x_i{y}_i(x_1-\overline{x})+\overline{y}(\sum x_i^2-n\overline{x}{x}_1))\\ ⋮\\ \frac{1}{t}(\sum x_i{y}_i(x_n-\overline{x})+\overline{y}(\sum x_i^2-n\overline{x}{x}_n))\end{array}\right]\end{array}$

   ​       其中，$t=\sum x_i^2-n{\overline{x}}^2$

   ​       代入$(x_1,y_1),(x_2,y_2)$得：

   ​       $\{\begin{array}{c}\hat{b}{x}_1+a=\frac{1}{t}(\sum x_i{y}_i(x_1-\overline{x})+\overline{y}(\sum x_i^2-n\overline{x}{x}_1)\\ \hat{b}{x}_2+a=\frac{1}{t}(\sum x_i{y}_i(x_2-\overline{x})+\overline{y}(\sum x_i^2-n\overline{x}{x}_2))\end{array}$

   ​       解得：$\{\begin{array}{c}\hat{b}=\frac{1}{t}(\sum x_i{y}_i-n\overline{x}\overline{y})=\frac{\sum x_i{y}_i-n\overline{x}\overline{y}}{\sum x_i^2-n{\overline{x}}^2}\\ \hat{a}=\overline{y}-\hat{b}\overline{x}\end{array}$

   ​    法二： 求拟合直线时，由上一讲可得$A^T\vec{b}=A^{T}A\vec{x}$

   ​       可以考虑这么计算：$A^T\left[\begin{array}{c}A|\vec{b}\end{array}\right]=\left[\begin{array}{c}{A}^{T}A|A^T\vec{b}\end{array}\right]$，再使用$A^T\vec{b}=A^{T}A\vec{x}$列出方程组得到$\vec{x}$

   ​       有$A^T\vec{b}=\left[\begin{array}{c}\sum x_i{y}_i\\ n\overline{y}\end{array}\right],A^{T}A=\left[\begin{array}{cc}\sum x_i^2& n\overline{x}\\ n\overline{x}& n\end{array}\right]$

   ​       所以可以列出的方程组为：$\{\begin{array}{c}\sum x_i^2\hat{b}+n\overline{x}\hat{a}=\sum x_i{y}_i\\ n\overline{x}\hat{b}+n\hat{a}=n\overline{y}\end{array}$，解得：$\{\begin{array}{c}\hat{b}=\frac{\sum x_i{y}_i-n\overline{x}\overline{y}}{\sum x_i^2-n{\overline{x}}^2}\\ \hat{a}=\overline{y}-\hat{b}\overline{x}\end{array}$

   例： 求$(1,1),(2,2),(3,2)$三点的一条拟合直线

   ​   设直线为$\hat{y}=\hat{b}x+\hat{a}$

   ​   有$\{\begin{array}{c}\hat{b}+\hat{a}=1\\ 2\hat{b}+\hat{a}=2\\ 3\hat{b}+\hat{a}=2\end{array}$，即$\begin{array}{cccc}\left[\begin{array}{cc}1& 1\\ 2& 1\\ 3& 1\end{array}\right]& \left[\begin{array}{c}\hat{b}\\ \hat{a}\end{array}\right]& =& \left[\begin{array}{c}1\\ 2\\ 2\end{array}\right]\\ A& \hat{\vec{x}}& & \vec{b}\end{array}$

   ​   易得$\vec{b}$不属于$A$的列空间

   ​   由$\begin{array}{cccc}\left[\begin{array}{ccc}1& 2& 3\\ 1& 1& 1\end{array}\right]& \left[\begin{array}{cccc}1& 1& |& 1\\ 2& 1& |& 2\\ 3& 1& |& 2\end{array}\right]& =& \left[\begin{array}{cccc}14& 6& |& 11\\ 6& 3& |& 5\end{array}\right]\\ A^T& \left[\begin{array}{c}A|\vec{b}\end{array}\right]& & \left[\begin{array}{c}{A}^{T}A|A^T\vec{b}\end{array}\right]\end{array}$可得：$\{\begin{array}{c}14\hat{b}+6\hat{a}=11\\ 6\hat{b}+3\hat{a}=5\end{array}$，解得：$\{\begin{array}{c}\hat{b}=\frac{1}{2}\\ \hat{a}=\frac{2}{3}\end{array}$

   ​   所以$\hat{y}=\frac{1}{2}x+\frac{2}{3}$

---

打赏

制作不易，若有帮助，欢迎打赏！