双变量回归模型：估计问题

普通最小二乘估计

$$min\sum (Y_i-{\hat{Y}}_i{)}^2=\sum (Y_i-{\beta }_0-{\beta }_1{X}_i{)}^2$$
对${\beta }_0,{\beta }_1$求导，令为0:
$$\begin{gathered} 2\sum (Y_i-{\beta }_0-{\beta }_1{X}_i)(-1)=0 \\ 2\sum (Y_i-{\beta }_0-{\beta }_1{X}_i)(-X_i)=0 \end{gathered}$$
即
$$\begin{gathered} \sum Y_i=n{\beta }_0+{\beta }_1\sum X_i \\ \sum X_i{Y}_i={\beta }_0\sum X_i+{\beta }_1\sum X_i^2 \end{gathered}$$
上式乘以$\sum X_i$,下式乘以$n$得
$$\begin{gathered} \sum X_i\sum Y_i=n{\beta }_0\sum X_i+{\beta }_1(\sum X_i{)}^2 \\ n\sum X_i{Y}_i=n{\beta }_0\sum X_i+n{\beta }_1\sum X_i^2 \end{gathered}$$
两式相减得
$$\sum X_i\sum Y_i-n\sum X_i{Y}_i={\beta }_1[(\sum X_i{)}^2-n\sum X_i^2]$$
解得
$$\begin{gathered} {\hat{\beta }}_1=\frac{n\sum X_i{Y}_i-\sum X_i\sum Y_i}{n\sum X_i^2-(\sum X_i{)}^2} \\ =\frac{\sum (X_i-\overline{X})(Y_i-\overline{Y})}{\sum (X_i-\overline{X}{)}^2}=\frac{\sum x_i{y}_i}{\sum x_i^2} \end{gathered}$$
而
$${\hat{\beta }}_0=\overline{Y}-{\hat{\beta }}_1\overline{X}$$

估计的方差

• ${\hat{\beta }}_1$的方差

$$\begin{gathered} var({\hat{\beta }}_1)=E({\hat{\beta }}_1-E({\hat{\beta }}_1){)}^2 \\ =E(\sum \frac{x_i}{\sum x_i^2}{Y}_i-{\beta }_1{)}^2 \\ =E[\sum \frac{x_i}{\sum x_i^2}({\beta }_0+{\beta }_1{X}_i+{\mu }_i)-{\beta }_1{]}^2 \\ =E(\sum \frac{x_i}{\sum x_i^2}{\mu }_i{)}^2 \end{gathered}$$
由于$E({\mu }_i^2)=\sigma ,E({\mu }_i{\mu }_j)=0$,则上式为
$$var({\hat{\beta }}_1)={\sigma }^2\sum (\frac{x_i}{\sum x_i^2}{)}^2=\frac{{\sigma }^2}{\sum x_i^2}$$

• ${\beta }_0$的方差
  $$\begin{gathered} var({\hat{\beta }}_0)=E[\overline{Y}-\overline{X}\sum k_i({\beta }_0+{\beta }_1{X}_i+{\mu }_i)-{\beta }_0] \\ =E[\overline{Y}-{\beta }_1\overline{X}-\overline{X}\sum k_i{\mu }_i-{\beta }_0] \\ =E[\overline{X}\sum \frac{x_i}{\sum x_i^2}{\mu }_i{]}^2={\sigma }^2\sum (\overline{X}\frac{x_i}{\sum x_i}{)}^2 \\ ={\sigma }^2\frac{X_i^2}{n\sum x_i^2} \end{gathered}$$
• ${\sigma }^2$的估计
  $$\begin{gathered} Y_i={\beta }_0+{\beta }_1{X}_i+{\mu }_i \\ \overline{Y}={\beta }_0+{\beta }_1\overline{X}+\overline{u} \end{gathered}$$
  两式相减得
  $$y_i={\beta }_1{x}_i+({\mu }_i-\overline{\mu })$$
  又知道
  $$\hat{\mu }=y_i-{\hat{\beta }}_1{x}_i$$
  则
  $$\begin{gathered} \hat{\mu }={\beta }_1{x}_i+({\mu }_i-\overline{\mu })-{\hat{\beta }}_1{x}_i \\ =({\beta }_1-{\hat{\beta }}_1)x_i+({\mu }_i-\overline{\mu }) \end{gathered}$$
  那么
  $$\begin{gathered} E\sum {\hat{\mu }}^2=E[\sum ({\beta }_1-{\hat{\beta }}_1{)}^2{x}_i^2+\sum ({\mu }_i-\overline{\mu }{)}^2-2\sum ({\beta }_1-{\hat{\beta }}_1)x_i({\mu }_i-\overline{\mu })] \\ =var({\hat{\beta }}_1)\sum x_i^2+(n-1){\sigma }^2-2E[\sum k_i{\mu }_i{x}_i({\mu }_i-\overline{\mu })] \\ ={\sigma }^2+(n-1){\sigma }^2-2{\sigma }^2=(n-2){\sigma }^2 \end{gathered}$$
  那么令
  $${\hat{\sigma }}^2=\frac{\sum {\hat{\mu }}^2}{n-2}$$
  则$$E({\hat{\sigma }}^2)=(n-2){\sigma }^2/(n-2)={\sigma }^2$$
  说明是${\sigma }^2$的无偏估计。

  高斯-马尔可夫定理

  在给定经典假设下，OLS估计量是最优线性无偏估计量。
  • 无偏
    $$\begin{gathered} {\hat{\beta }}_1=\frac{\sum x_i{y}_i}{\sum x_i^2}=\sum \frac{x_i}{\sum x_i^2}{Y}_i \\ =\sum k_i{Y}_i=\sum k_i({\beta }_0+{\beta }_1{X}_i+{\mu }_i) \end{gathered}$$
    其中$k_i=x_i/\sum x_i^2$,有如下性质：
    $\sum k_i=0$;
    $\sum k_i{X}_i=\sum k_i{x}_i=1$；
    则
    $$E({\hat{\beta }}_1)=E\sum k_i{\beta }_0+E\sum k_i{X}_i{\beta }_1+E\sum k_i{\mu }_i={\beta }_1$$
    则${\hat{\beta }}_1$是${\beta }_1$的无偏估计量。
    $$\begin{gathered} {\hat{\beta }}_0=\overline{Y}-\sum k_i{Y}_i\overline{X}={\beta }_0+{\beta }_1\overline{X}-\overline{X}\sum k_i({\beta }_0+{\beta }_1{X}_i+{\mu }_i) \\ ={\beta }_0+{\beta }_1\overline{X}-\overline{X}{\beta }_1-\overline{X}\sum k_i{\mu }_i \\ ={\beta }_0-\overline{X}\sum k_i{\mu }_i \end{gathered}$$
    取期望
    $$E({\hat{\beta }}_0)={\beta }_0+E(\overline{X}\sum k_i{\mu }_i)={\beta }_0$$
    ${\hat{\beta }}_0$也是${\beta }_0$的无偏估计。
  • 有效
    设另一估计量
    $${\beta }_1^{\ast }=\sum w_i{Y}_i$$
    期望为$$E({\beta }_1^{\ast })={\beta }_0\sum w_i+{\beta }_1\sum w_i{X}_i$$
    若想无偏，必须有
    $\sum w_i=0$;
    $\sum w_i{X}_i=\sum w_i{x}_i=1$;
    方差为
    $$\begin{gathered} var({\beta }_1^{\ast })=\sum w_i^{2}var(Y_i) \\ ={\sigma }^2\sum [(w_i-k_i)+k_i{]}^2={\sigma }^2\sum [(w_i-k_i{)}^2+k_i^2-2(w_i{k}_i-k_i^2)] \\ ={\sigma }^2(w_i-k_i{)}^2+{\sigma }^2\frac{1}{\sum x_i^2} \end{gathered}$$
    只有当$w_i=k_i$时，方差才最小。