最小二乘估计理论

模型

不需输入输出为随机过程，求最优权向量使得输出估计结果的样本均方误差最小
b ^ ⃗ H = w ⃗ H A H { b ^ H = [ d ^ ( M ) d ^ ( M + 1 ) ⋯ d ^ ( N ) ] A H = [ u ( M ) u ( M + 1 ) ⋯ u ( N ) ] = [ u ( M ) u ( M + 1 ) ⋯ u ( N ) u ( M − 1 ) u ( M ) ⋯ u ( N − 1 ) ⋮ ⋮ ⋱ ⋮ u ( 1 ) u ( 2 ) ⋯ u ( N − M + 1 ) ] w = [ w 0 w 1 ⋯ w M − 1 ] H m i n { J } { e = b − b ^ J = e H e \begin {align} &\vec{\hat{b}}^H=\vec{w}^HA^H \\ &\left\{\begin{aligned} \hat{b}^H&=\begin{bmatrix} \hat{d}(M)&\hat{d}(M+1)&\cdots&\hat{d}(N) \end{bmatrix} \\ A^H&=\begin{bmatrix} u(M)&u(M+1)&\cdots&u(N)\end{bmatrix}\\ &=\begin{bmatrix} u(M)&u(M+1)&\cdots&u(N)\\ u(M-1)&u(M)&\cdots&u(N-1)\\ \vdots&\vdots&\ddots&\vdots\\ u(1)&u(2)&\cdots&u(N-M+1)\\ \end{bmatrix}\\ w&=\begin{bmatrix} w_0 & w_1 &\cdots &w_{M-1} \end{bmatrix} ^H\\ \end{aligned} \right.\\ &min\{J\}\\ &\left\{\begin{aligned} &e=b-\hat{b} \\ &J=e^He \end{aligned} \right. \end {align} ​b^ H=w HAH⎩ ⎨ ⎧​b^HAHw​=[d^(M)​d^(M+1)​⋯​d^(N)​]=[u(M)​u(M+1)​⋯​u(N)​]= ​u(M)u(M−1)⋮u(1)​u(M+1)u(M)⋮u(2)​⋯⋯⋱⋯​u(N)u(N−1)⋮u(N−M+1)​ ​=[w0​​w1​​⋯​wM−1​​]H​min{J}{​e=b−b^J=eHe​​​

方法

最优权向量满足确定性正则方程：
$$A^{H}A\hat{w}=A^{H}b$$

算法

基于SVD的LS算法

LS最优权重仅与数据矩阵$A$的奇异值和特征向量有关，故求得数据矩阵的奇异值和特征向量即可求得最有权重。
$$\begin{array}{r}\{\begin{array}{rl}& \hat{w}=\sum _{i=1}^K(\frac{{\overrightarrow{x_i}}^H\vec{\theta }}{{\sigma }_i^2})\overrightarrow{x_i}\\ & \vec{\theta }=A^{H}b\end{array}\end{array}$$
关于$A^{H}A$是否奇异分类讨论：
$$\begin{array}{r}\{\begin{array}{rl}非奇异K=M:& \hat{w}=\sum _{i=1}^M(\frac{{\overrightarrow{x_i}}^H\vec{\theta }}{{\sigma }_i^2})\overrightarrow{x_i}\\ 奇异K<M:& \hat{w}=\sum _{i=1}^K(\frac{{\overrightarrow{x_i}}^H\vec{\theta }}{{\sigma }_i^2})\overrightarrow{x_i}(加入最小范数条件使得有唯一解)\end{array}\end{array}$$