【学习笔记】数理统计习题十一

Q1: Consider the linear model

$$y_i={\beta }_0+{\beta }_1{x}_i+{ϵ}_i,{ϵ}_i\stackrel{iid}{\sim }N(0,{\sigma }^2),i=1,\dots ,n.$$

1. Derive the maximum likelihood estimators (MLE) for ${\beta }_0,{\beta }_1$. Are they consistent with the least square estimators (LSE)?

2. Derive the MLE for ${\sigma }^2$ and look at its unbiasedness.

3. A very slippery point is whether to treat the $x_i$ as fixed numbers or as random variables. In the class, we treated the predictors $x_i$ as fixed numbers for sake of convenience. Now suppose that the predictors $x_i$ are iid random variables (independent of ${ϵ}_i$) with density $f_X(x;\theta )$ for some parameter $\theta$. Write down the likelihood function for all of our data $(x_i,y_i),i=1,\dots ,n$. Derive the MLE for ${\beta }_0,{\beta }_1$ and see whether the MLE changes if we work with the setting of random predictors?

解： 注意到$y_i\sim N({\beta }_0+{\beta }_1{x}_i,{\sigma }^2)$是独立的，则似然函数为$$L({\beta }_0,{\beta }_1,{\sigma }^2)=\prod _{i=1}^n\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2}{2{\sigma }^2}}=(2\pi {\sigma }^2{)}^{-n/2}{e}^{-\frac{Q({\beta }_0,{\beta }_1)}{2{\sigma }^2}}$$其中$Q({\beta }_0,{\beta }_1)={\sum }_{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2$，对于给定的${\sigma }^2$，为了使$L({\beta }_0,{\beta }_1,{\sigma }^2)$最大化，则需要使$Q({\beta }_0,{\beta }_1)$最小化，可知这与最小二乘估计量的估计方法一致，因此，最大似然估计值与最小二乘估计量一样。
也就是说
$${\hat{\beta }}_1=\frac{{\ell }_{xy}}{{\ell }_{xx}}=\frac{{\sum }_{i=1}^n(y_i-\bar{y})(x_i-\bar{x})}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2},{\hat{\beta }}_0=\bar{y}-{\hat{\beta }}_1\bar{x}$$接下来我们选择${\sigma }^2$，令似然函数$L({\hat{\beta }}_0,{\hat{\beta }}_1,{\sigma }^2)=(2\pi {\sigma }^2{)}^{-n/2}{e}^{-\frac{Q({\hat{\beta }}_0,{\hat{\beta }}_1)}{2{\sigma }^2}}$最大化，容易得到这样的${\sigma }^2$为$${\hat{\sigma }}_{MLE}^2=\frac{Q({\hat{\beta }}_0,{\hat{\beta }}_1)}{n}=\frac{S_e^2}{n}$$我们已经证明了$E[S_e^2]=(n-2){\sigma }^2$，因此$E[{\hat{\sigma }}_{MLE}^2]=\frac{n-2}{n}{\sigma }^2$，所以这不是${\sigma }^2$的无偏估计。
如果$x_i$是密度函数为$f_X(x;\theta )$的随机变量，则关于$(x_i,y_i)$的似然函数为$$\begin{array}{rl}L({\beta }_0,{\beta }_1,{\sigma }^2,\theta )& =\prod _{i=1}^n{f}_X(x_i;\theta )f(y_i|x_i)\\ & =\prod _{i=1}^n[f_X(x_i;\theta )\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2}{2{\sigma }^2}}]\\ & =(2\pi {\sigma }^2{)}^{-n/2}{e}^{-\frac{Q({\beta }_0,{\beta }_1)}{2{\sigma }^2}}{f}_X(x_i;\theta )\end{array}$$对于固定的${\sigma }^2,\theta$，为了最大化$L({\beta }_0,{\beta }_1,{\sigma }^2,\theta )$，则需要使$Q({\beta }_0,{\beta }_1)$最小化，因此最大似然估计值没有发生变化。

Q2: Consider the linear model without intercept

$$y_i=\beta x_i+{ϵ}_i,i=1,\dots ,n,$$

where ${ϵ}_i$ are independent with $E[{ϵ}_i]=0$ and $Var[{ϵ}_i]={\sigma }^2$.

• Write down the least square estimator $\hat{\beta }$ for $\beta$, and derive an unbiased estiamtor for ${\sigma }^2$.

• For fixed $x_0$, let ${\hat{y}}_0=\hat{\beta }{x}_0$. Work out $Var[{\hat{y}}_0]$.

解： 令$Q(\beta )={\sum }_{i=1}^n(y_i-\beta x_i{)}^2$，不难发现，最小值点$\hat{\beta }$满足
$$\frac{\partial Q}{\partial \beta }=-2\sum _{i=1}^n(y_i-\beta x_i)x_i=0$$ 于是得到最小二乘估计量：
$$\hat{\beta }=\frac{{\sum }_{i=1}^n{x}_i{y}_i}{{\sum }_{i=1}^n{x}_i^2}$$注意到 E [ Q ( β ^ ) ] = E [ ∑ i = 1 n y i 2 + β ^ 2 ∑ i = 1 n x i 2 − 2 β ^ ∑ i = 1 n x i y i ] = ∑ i = 1 n { V a r [ y i ] + ( E [ y i ] ) 2 } − E [ β ^ ∑ i = 1 n x i y i ] = ∑ i = 1 n ( σ 2 + β 2 x i 2