最大似然估计和加权最大似然估计的渐进正态性

最大似然估计的渐近分布

记似然函数为
$$L(\theta )=\prod _{i=1}^{n}f(X_i;\theta )$$
令$l(\theta )=logL(\theta )$为对数似然函数，设$\theta$为真值，$\hat{\theta }$为最大似然估计值。则有
$$\frac{\partial l(\hat{\theta })}{\partial \theta }=\frac{\partial l(\theta )}{\partial \theta }+\frac{{\partial }^{2}l(\theta )}{\partial {\theta }^2}(\hat{\theta }-\theta )=0$$
从而
$$\sqrt{n}(\hat{\theta }-\theta )=-\sqrt{n}\frac{l^{\prime }(\theta )}{l^{\prime \prime }(\theta )}=\frac{(1/\sqrt{n})l^{\prime }(\theta )}{-(1/n)l^{\prime \prime }(\theta )}$$
(i)由于
$$\begin{array}{rl}\frac{1}{\sqrt{n}}{l}^{\prime }(\theta )& =\frac{1}{\sqrt{n}}\sum _i\frac{\partial logf(X_i;\theta )}{\partial \theta }\\ & =\sqrt{n}\frac{1}{n}\sum _i\frac{\partial logf(X_i;\theta )}{\partial \theta }\end{array}$$
由于$\mathbb{E}[\partial logf(X_i;\theta )/\partial \theta =/0]$以及$\mathbb{V}[\partial logf(X_i;\theta )/\partial \theta ]=I(\theta )$（看我之前Fisher信息矩阵的博客）,进而由中心极限定理，可知
$$\frac{1}{n}\sum _i\frac{\partial logf(X_i;\theta )}{\partial \theta }⇝N(0,I(\theta )/n)$$
因此
$$\sqrt{n}\frac{1}{n}\sum _i\frac{\partial logf(X_i;\theta )}{\partial \theta }⇝N(0,I(\theta ))$$
(ii)由于
$$-(1/n)l^{\prime \prime }(\theta )=-\frac{1}{n}\sum _i\frac{\partial lo{g}^{2}f(X_i;\theta )}{\partial {\theta }^2}$$
由于$E[-\frac{\partial lo{g}^{2}f(X_i;\theta )}{\partial {\theta }^2}]=I(\theta )$，因此
$$-(1/n)l^{\prime \prime }(\theta )\to I(\theta )$$

$$\frac{(1/\sqrt{n})l^{\prime }(\theta )}{-(1/n)l^{\prime \prime }(\theta )}⇝N(0,\frac{I(\theta )}{I(\theta {)}^2})=N(0,I(\theta {)}^{-1})$$

因此，最大似然估计量具有渐进正态分布。

下面我们证明加权最大似然估计具有渐进正态分布。

Theorem 1.(Hidetoshi, 2000)

在一定的正则条件下，即模型足够光滑等，设加权最小二乘估计器为$\theta$，真实值为${\theta }^{\ast }$，则$\sqrt{n}(\theta -{\theta }^{\ast })$的渐进正态分布为$N(0,H^{-1}G{H}^{-1})$，其中，$H$和$G$均为$m\times m$非奇异矩阵，定义为
$$G=E[\frac{\partial l_w(x,y|\theta )}{\partial \theta }{|}_{{\theta }^{\ast }}\frac{\partial l_w(x,y|\theta )}{\partial {\theta }^T}{|}_{{\theta }^{\ast }}]$$

$$H=E[\frac{{\partial }^2{l}_w(x,y|\theta )}{\partial \theta \partial {\theta }^T}{|}_{{\theta }^{\ast }}]$$

其中，
$$l_w(x,y|\theta )=-w(x)logp(y|x,\theta )$$

Proof.

证明思路应该是与最大似然估计类似。

最大加权似然估计量满足
$$\sum _i\frac{\partial l_w(x_i,y_i|\theta )}{\partial \theta }{|}_{\theta =\theta \ast }=0$$
求导，有
$$\sum _i\frac{\partial l_w(x_i,y_i|\theta )}{\partial \theta }{|}_{\theta =\theta \ast }+\sum _i\frac{{\partial }^2{l}_w(x_i,y_i|\theta )}{\partial \theta \partial {\theta }^{\prime }}{|}_{\theta ={\theta }^{\ast }}(\theta -{\theta }^{\ast })=0$$
进一步
$$n^{\frac{1}{2}}(\theta -{\theta }^{\ast })=\frac{n^{-1/2}}{n^{-1}}\frac{{\sum }_i\frac{\partial l_w(x_i,y_i|\theta )}{\partial \theta }{|}_{\theta =\theta \ast }}{{\sum }_i\frac{{\partial }^2{l}_w(x_i,y_i|\theta )}{\partial \theta \partial {\theta }^{\prime }}{|}_{\theta ={\theta }^{\ast }}}$$
变形：
$$n^{-1}\sum _i\frac{{\partial }^2{l}_w(x_i,y_i|\theta )}{\partial \theta \partial {\theta }^{\prime }}{|}_{\theta ={\theta }^{\ast }}{n}^{\frac{1}{2}}(\theta -{\theta }^{\ast })=n^{-1/2}\sum _i\frac{\partial l_w(x_i,y_i|\theta )}{\partial \theta }{|}_{\theta =\theta \ast }$$
根据中心极限定理，右侧$⇝N(0,G)$，而左侧依据概率收敛到$H\sqrt{n}(\theta -{\theta }^{\ast })$,从而直接得到结论
$$\sqrt{n}(\theta -{\theta }^{\ast })⇝N(0,H^{-1}G{H}^{-1})$$