多元线性回归

多元线性回归

$$y=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_n\end{array}\right],X=\left[\begin{array}{cccc}1& x_{11}& \cdots & x_{1p}\\ 1& x_{21}& \cdots & x_{2p}\\ ⋮& ⋮& & ⋮\\ 1& x_{n1}& \cdots & x_{np}\end{array}\right],ϵ=\left[\begin{array}{c}{ϵ}_1\\ {ϵ}_2\\ ⋮\\ {ϵ}_n\end{array}\right],\beta =\left[\begin{array}{c}{\beta }_0\\ {\beta }_1\\ ⋮\\ {\beta }_p\end{array}\right]$$

$$\mathbf{y}=\mathbf{X}\mathbf{\beta }+\epsilon$$

$$\mathbf{X}=\left(1,{\mathbf{x}}_1,\dots ,{\mathbf{x}}_p\right)--n\times (p+1)$$

$$\epsilon ={\left({\epsilon }_1,\dots ,{\epsilon }_n\right)}^{\prime }$$

Gauss-Markov条件:
$$\{\begin{array}{l}E\left({\epsilon }_i\right)=0,i=1,\dots ,n\\ \mathrm{Cov}\left({\epsilon }_i,{\epsilon }_j\right)=0,i\ne j;\mathrm{Var}\left({\epsilon }_i\right)={\sigma }^2\end{array}$$

正态性假设:
$$\{\begin{array}{l}{\epsilon }_i\sim N\left(0,{\sigma }^2\right),i=1,\dots ,n\\ {\epsilon }_1,\dots ,{\epsilon }_n\text{ 相互独立 }\end{array}$$

LSE

$$Q(\mathbf{\beta })=(\mathbf{y}-\mathbf{X}\mathbf{\beta }{)}^{\prime }(\mathbf{y}-\mathbf{X}\mathbf{\beta })$$

$$\frac{\partial Q(\mathbf{\beta })}{\partial \mathbf{\beta }}=-{\mathbf{X}}^{\prime }2(\mathbf{y}-\mathbf{X}\mathbf{\beta })=-2{\mathbf{X}}^{\prime }(\mathbf{y}-\mathbf{X}\mathbf{\beta })=0$$

$$\hat{\mathbf{\beta }}={\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\prime }\mathbf{y}$$

$$\hat{\mathbf{y}}=\mathbf{X}\hat{\mathbf{\beta }}=\mathbf{X}{\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\prime }\mathbf{y}\stackrel{\text{def}}{=}\mathbf{H}\mathbf{y}$$

$H=X(X^{\prime }X{)}^{-1}{X}^{\prime }\to H^2=X(X^{\prime }X{)}^{-1}{X}^{\prime }\ast X(X^{\prime }X{)}^{-1}{X}^{\prime }=X(X^{\prime }X{)}^{-1}{X}^{\prime }=H$

$(I_n-H{)}^2=I_n^2-2H{I}_n+H^2=I_n-H$

Resid

$$\mathbf{e}=\mathbf{y}-\hat{\mathbf{y}}\text{ 为 }\mathbf{y}\text{ 的残差向量，将 }\hat{\mathbf{y}}=\mathbf{H}\mathbf{y}\text{ 代入 }$$

$$\mathbf{e}=\mathbf{y}-\mathbf{H}\mathbf{y}=(\mathbf{I}-\mathbf{H})\mathbf{y}$$

$$\begin{array}{rl}D(\mathbf{e})& =\mathrm{Cov}(\mathbf{e},\mathbf{e})\\ & =\mathrm{Cov}((\mathbf{I}-\mathbf{H})\mathbf{y},(\mathbf{I}-\mathbf{H})\mathbf{y})\\ & =(\mathbf{I}-\mathbf{H})\mathrm{Cov}(\mathbf{y},\mathbf{y})(\mathbf{I}-\mathbf{H}{)}^{\prime }\\ & =(\mathbf{I}-\mathbf{H}){\sigma }^2\mathbf{I}(\mathbf{I}-\mathbf{H}{)}^{\prime }\\ & ={\sigma }^2(\mathbf{I}-\mathbf{H})\end{array}$$

$$\mathrm{Var}\left(e_i\right)=\left(1-h_{ii}\right){\sigma }^2,i=1,\dots ,n$$

$\hat{\beta }$为无偏估计
$$\begin{array}{rl}E(\hat{\mathbf{\beta }})& =E\left\{{\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\prime }\mathbf{y}\right\}\\ & ={\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\prime }E\mathbf{y}\\ & ={\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\prime }\mathbf{X}\mathbf{\beta }=\mathbf{\beta }\end{array}$$

$$D(\hat{\mathbf{\beta }})={\sigma }^2{\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}$$

---

exer

$$Proof:{\hat{\sigma }}^2=\frac{1}{n-p-1}\sum _{i=1}^n{e}_i^2=\frac{1}{n-p-1}{\mathbf{e}}^{\prime }\mathbf{e}$$

$$E\left(\sum _{i=1}^n{e}_i^2\right)=\sum _{i=1}^{n}D\left(e_i\right)$$

$$E\left(\sum _{i=1}^n{e}_i^2\right)=\sum _{i=1}^{n}D\left(e_i\right)=\sum _{i=1}^n{\sigma }^2\left(1-h_{ii}\right)={\sigma }^2\sum _{i=1}^n\left(1-h_{ii}\right)={\sigma }^2\left(n-\sum _{i=1}^n{h}_{ii}\right)={\sigma }^2(n-p-1)$$

---

$$\begin{array}{l}\mathrm{diag}(\mathbf{H})=\left(h_{11},\dots ,h_{nn}\right)\\ \mathrm{tr}(H)={\sum }_{i=1}^n{h}_{ii}=\mathrm{tr}\left(\mathbf{X}{\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\prime }\right)=\mathrm{tr}\left({\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\prime }\mathbf{X}\right)=p+1\end{array}$$

$$计算\mathrm{Cov}(\mathbf{e},\hat{\mathbf{\beta }})\to {\hat{\sigma }}^2与\hat{\beta }的独立性$$

$$\begin{array}{l}\mathrm{cov}(\hat{\beta },e)=\mathrm{cov}\left({\left(X^{T}X\right)}^{-1}{X}^{T}y,(I-H)y\right)={\left(X^{T}X\right)}^{-1}{X}^T\mathrm{cov}(y,y)(I-H)\\ {\sigma }^2{\left(X^{T}X\right)}^{-1}{X}^T(I-H)=0\end{array}$$

$$\text{ 这是因为最小二乘法 }(I-H)X=0\text{ 可以推出 }{X}^T(I-H)=0$$

$$\begin{array}{rl}& 在正态分布假定下，对一元线性回归模型，构造假设检验H_0:2{\beta }_0={\beta }_1\text{ v.s. }{H}_1:2{\beta }_0\ne {\beta }_1\text{ 的检验统计量}\end{array}$$

---

在$G-M$假设下，最小二乘估计$\hat{\mathbf{\beta }}$与残差向量$e$不相美，即 $Cov(\hat{\mathbf{\beta }},\mathbf{e})=0$ ,进一步，在正态假设下，$\hat{\mathbf{\beta }}$与e独立，从而$\hat{\mathbf{\beta }}$与$SSE={\mathbf{e}}^{\prime }\mathbf{e}=\Vert \mathbf{e}{\Vert }^2$ 独立

---

当$y\sim N\left(\mathbf{X}\mathbf{\beta },{\sigma }^2{\mathbf{I}}_n\right)$ 时，则
$$(1)\hat{\mathbf{\beta }}\sim N\left(\mathbf{\beta },{\sigma }^2{\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}\right)$$

$$(2)\mathrm{SSE}/{\sigma }^2\sim {\chi }^2(n-p-1)$$