【概率论与数理统计(研究生课程)】知识点总结9(回归分析)

原文地址：【概率论与数理统计(研究生课程)】知识点总结9(回归分析)

一元线性回归模型

$$\begin{array}{rl}& y={\beta }_0+{\beta }_{1}x+ϵ,ϵ\sim N(\mu ,{\sigma }^2)\\ & E(ϵ)=0,D(ϵ)={\sigma }^2>0⟹E(y)={\beta }_0+{\beta }_{1}x\end{array}$$

回归方程：$\hat{y}=\widehat{{\beta }_0}+\widehat{{\beta }_1}x$

推导过程：
$$\begin{array}{rl}{y}_i-E(y_i)& =y_i-({\beta }_0+{\beta }_1{x}_i)\\ Q({\beta }_1,{\beta }_2)& =\sum _{i=1}^n(y_i-E(y_i){)}^2\\ & =\sum _{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2\\ \text{make }\frac{\partial Q({\beta }_0,{\beta }_1)}{\partial {\beta }_0}& =-2\sum _{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i)=0\\ \text{make }\frac{\partial Q({\beta }_0,{\beta }_1)}{\partial {\beta }_1}& =-2\sum _{i=1}^n{x}_i(y_i-{\beta }_0-{\beta }_1{x}_i)=0\end{array}$$
整理得到正规方程组：
$$\begin{array}{rl}& n\widehat{{\beta }_0}+n\bar{x}\widehat{{\beta }_1}=n\bar{y}(1)\\ & n\bar{x}\widehat{{\beta }_0}+(\sum _{i=1}^n{x}_i^2)\widehat{{\beta }_1}=\sum _{i=1}^n{x}_i{y}_i(2)\end{array}$$
解上述方程组得到：
$$\begin{array}{rl}& \widehat{{\beta }_1}=\frac{L_{xy}}{L_{xx}}\\ & \widehat{{\beta }_0}=\bar{y}-\widehat{{\beta }_1}\bar{x}\\ & L_{xx}=\sum _{i=1}^n(x_i-\bar{x}{)}^2=\sum _{i=1}^n{x}_i^2-n{\bar{x}}^2=\sum _{i=1}^n{x}_i^2-\frac{1}{n}(\sum _{i=1}^n{x}_i{)}^2\\ & L_{yy}=\sum _{i=1}^n(y_i-\bar{y}{)}^2=\sum _{i=1}^n{y}_i^2-n{\bar{y}}^2=\sum _{i=1}^n{y}_i^2-\frac{1}{n}(\sum _{i=1}^n{y}_i{)}^2\\ & L_{xy}=\sum _{i=1}^n(x_i-\bar{x})(y_i-\bar{y})=\sum _{i=1}^n{x}_i{y}_i-n\bar{x}\bar{y}=\sum _{i=1}^n{x}_i{y}_i-\frac{1}{n}\sum _{i=1}^n{x}_i\sum _{i=1}^n{y}_i\end{array}$$

如果题目中给了$\sum$形式的数据， $L_{xx},L_{yy},L_{xy}$一般用上述公式最右边的方式来求。

残差/剩余平方和

$$Q_e=\sum _{i=1}^n{e}_i^2=\sum _{i=1}^n(y_i-\widehat{y_i}{)}^2=\sum _{i=1}^n(y_i-\widehat{{\beta }_0}-\widehat{{\beta }_1}{x}_i{)}^2=L_{yy}-\widehat{{\beta }_1}{L}_{xy}=L_{yy}-\frac{L_{xy}^2}{L_{xx}}$$

定理：$\frac{Q_e}{{\sigma }^2}\sim {\chi }^2(n-2)$
$$\begin{array}{rl}& E(\frac{Q_e}{{\sigma }^2})=n-2\\ ⟹& E(\frac{Q_e}{n-2})={\sigma }^2\\ ⟹& \widehat{{\sigma }^2}=\frac{Q_e}{n-2}\end{array}$$
${\hat{\sigma }}^2$的无偏估计为$\frac{Q_e}{n-2}$

最小二乘估计量的性质

${\beta }_0,{\beta }_1$的最小二乘估计量都是无偏的：$E(\widehat{{\beta }_0})={\beta }_0,E(\widehat{{\beta }_1})={\beta }_1$

$\widehat{{\beta }_0}\sim N({\beta }_0,(\frac{1}{n}+\frac{{\bar{x}}^2}{L_{xx}}){\sigma }^2)$

$\widehat{{\beta }_1}\sim N({\beta }_1,\frac{{\sigma }^2}{L_{xx}})$

$Cov(\widehat{{\beta }_0},\widehat{{\beta }_1})=-\frac{\bar{x}}{L_{xx}}{\sigma }^2$

$\widehat{y_0}\sim N({\beta }_0+{\beta }_1{x}_0,(\frac{1}{n}+\frac{(x_0-\bar{x}{)}^2}{L_{xx}}){\sigma }^2)$

回归方程显著性检验（t、F、r）

1. 提出原假设和备择假设（回归方程是否显著，反映在斜率是否为0）：

$$H_0:{\beta }_1=0;H_1:{\beta }_1\ne 0$$

2. 选取统计量：
   $$\begin{array}{rl}& \widehat{{\beta }_1}\sim N({\beta }_1,\frac{{\sigma }^2}{L_{xx}})\\ ⟹& \frac{\widehat{{\beta }_1}-{\beta }_1}{\sqrt{\frac{{\sigma }^2}{L_{xx}}}}\sim N(0,1)\\ \stackrel{H_0}{\to }& \frac{\widehat{{\beta }_1}\sqrt{L_{xx}}}{\sigma }\sim N(0,1)\end{array}$$
   若需构造$t$检验，还需要一个${\chi }^2$分布，而$\frac{Q_e}{{\sigma }^2}\sim {\chi }^2(n-2)$，从而：
   $$T=\frac{\frac{\widehat{{\beta }_1}\sqrt{L_{xx}}}{\sigma }}{\sqrt{\frac{Q_e}{{\sigma }^2}/(n-2)}}\stackrel{\widehat{{\sigma }^2}=\frac{Q_e}{n-2}}{\to }\frac{\widehat{{\beta }_1}\sqrt{L_{xx}}}{\hat{\sigma }}\sim t(n-2)$$
   若使用$F$检验，需要计算回归平方和以及残差平方和：
   $$\begin{array}{rl}& S_R^2=\sum _{i=1}^n(\widehat{y_i}-\overline{y_i}{)}^2=\widehat{{\beta }_1}{L}_{xy}\\ & S_e^2=\sum _{i=1}^n(y_i-\widehat{y_i}{)}^2=S_T^2-S_R^2=L_{yy}-\widehat{{\beta }_1}{L}_{xy}\\ & \frac{S_R^2}{{\sigma }^2}\sim {\chi }^2(1),\frac{S_e^2}{{\sigma }^2}\sim {\chi }^2(n-2)\\ & F=\frac{\frac{S_R^2}{{\sigma }^2}/1}{\frac{S_e^2}{{\sigma }^2}/(n-2)}=\frac{(n-2)S_R^2}{S_e^2}\sim F(1,n-2)\end{array}$$

3. 拒绝域

   $t$检验拒绝域：$|T|=|\frac{\widehat{{\beta }_1}\sqrt{L_{xx}}}{\hat{\sigma }}|\ge t_{\frac{\alpha }{2}}(n-2)$

   $F$检验拒绝域：$F\ge F_{\alpha }(1,n-2)$

4. 确定$t_{\frac{\alpha }{2}(n-2)}or{F}_{\alpha }(1,n-2)$

5. 计算$|T|orF$

6. 判断结果

回归系数的区间估计

$$\begin{array}{rl}& \widehat{{\beta }_1}\sim N({\beta }_1,\frac{{\sigma }^2}{L_{xx}})\\ ⟹& \frac{\widehat{{\beta }_1}-{\beta }_1}{\sqrt{\frac{{\sigma }^2}{L_{xx}}}}\sim N(0,1)\\ ⟹& \frac{(\widehat{{\beta }_1}-{\beta }_1)\sqrt{L_{xx}}}{\sigma }\sim N(0,1)\\ T=\frac{\frac{(\widehat{{\beta }_1}-{\beta }_1)\sqrt{L_{xx}}}{\sigma }}{\sqrt{\frac{Q_e}{{\sigma }^2}/(n-2)}}& \stackrel{\widehat{{\sigma }^2}=\frac{Q_e}{n-2}}{\to }\frac{(\widehat{{\beta }_1}-{\beta }_1)\sqrt{L_{xx}}}{\hat{\sigma }}\sim t(n-2)\end{array}$$

则${\beta }_1$置信水平为$1-\alpha$的置信区间为：$(\widehat{{\beta }_1}\pm \frac{\hat{\sigma }}{\sqrt{L_{xx}}}{t}_{\frac{\alpha }{2}}(n-2))$

估计

设回归方程为$\hat{y}=\widehat{{\beta }_0}+\widehat{{\beta }_1}x$，对任意给定的$x=x_0$，$y_0$的均值$E(y_0)={\beta }_0+{\beta }_1{x}_0$，$E(y_0)$的无偏估计为$\widehat{y_0}=\widehat{{\beta }_0}+\widehat{{\beta }_1}{x}_0$

$\widehat{{\beta }_0}\sim N({\beta }_0,(\frac{1}{n}+\frac{{\bar{x}}^2}{L_{xx}}){\sigma }^2)$

$\widehat{{\beta }_1}\sim N({\beta }_1,\frac{{\sigma }^2}{L_{xx}})$

$Cov(\widehat{{\beta }_0},\widehat{{\beta }_1})=-\frac{\bar{x}}{L_{xx}}{\sigma }^2$

$D(\widehat{y_0})=D(\widehat{{\beta }_0})+D(\widehat{{\beta }_1}{x}_0)+2Cov(\widehat{{\beta }_0},\widehat{{\beta }_1}{x}_0)=(\frac{1}{n}+\frac{(\bar{x}-x_0{)}^2}{L_{xx}}){\sigma }^2$

$\widehat{y_0}\sim N({\beta }_0+{\beta }_1{x}_0,(\frac{1}{n}+\frac{(x_0-\bar{x}{)}^2}{L_{xx}}){\sigma }^2)$
于是$E(y_0)$的置信度为$1-\alpha$的置信区间为：
$$(\widehat{y_0}-{\delta }_0,\widehat{y_0}+{\delta }_0),\delta =t_{\frac{\alpha }{2}}(n-2)\hat{\sigma }\sqrt{\frac{1}{n}+\frac{(x_0-\bar{x}{)}^2}{L_{xx}}}$$

区间预测

$$\begin{array}{r}{y}_0-\widehat{y_0}\sim N(0,[1+\frac{1}{n}+\frac{(x_0-\bar{x}{)}^2}{L_{xx}}]{\sigma }^2)\\ U=\frac{y_0-\widehat{y_0}}{\sigma \sqrt{1+\frac{1}{n}+\frac{(x_0-\bar{x}{)}^2}{L_{xx}}}}\sim N(0,1)\\ T=\frac{y_0-\widehat{y_0}}{\hat{\sigma }\sqrt{1+\frac{1}{n}+\frac{(x_0-\bar{x}{)}^2}{L_{xx}}}}\sim t(n-2)\end{array}$$

因此，$y_0$的置信度为$1-\alpha$的区间为
$$(\widehat{y_0}-\delta ,\widehat{y_0}+\delta ),\delta =t_{\frac{\alpha }{2}}(n-2)\hat{\sigma }\sqrt{1+\frac{1}{n}+\frac{(x_0-\bar{x}{)}^2}{L_{xx}}}$$

可线性化的一元非线性回归