Fisher信息与最大似然估计的渐进正态性（附有在Bernoulli分布上的计算）

写在前面

最大似然估计具有很多好的性质，包括相合性，同变性，渐进正态性等。本文主要关注的是渐进正态性。渐近正态性表明，估计量的极限分布是正态分布。而该正态分布的方差，与Fisher信息有着密不可分的关系。

Fisher信息

（定义）记分函数（Score Function):
$$s(X;\theta )=\frac{\partial logf(X;\theta )}{\partial \theta }.$$
（定义）Fisher信息量(Fisher Information):
$$\begin{array}{rl}{I}_n(\theta )& =\mathbb{V}(\sum _{i=1}^{n}s(X_i;\theta ))\\ & =\sum _{i=1}^n\mathbb{V}(s(X_i;\theta ))\end{array}$$
（定理）
$${\mathbb{E}}_{\theta }[s(X;\theta )]=0$$
证明：
$$\begin{array}{rl}{\mathbb{E}}_{\theta }[s(X;\theta )]& ={\int }_x\frac{\partial logf(x;\theta )}{\partial \theta }f(x;\theta )dx\\ & ={\int }_x\frac{1}{f(x;\theta )}\frac{\partial f(x;\theta )}{\partial \theta }f(x;\theta )dx\\ & ={\int }_x\frac{\partial f(x;\theta )}{\partial \theta }dx\\ & =\frac{\partial }{\partial \theta }{\int }_{x}f(x;\theta )dx=\frac{\partial }{\partial \theta }1\\ & =0\end{array}$$
（定理）若$f(X;\theta )$二阶可导，则Fisher信息矩阵可以写为如下形式：
$$I_n(\theta )=nI(\theta )=-n{\int }_x\frac{{\partial }^{2}logf(x;\theta )}{\partial {\theta }^2}f(x;\theta )dx$$
证明：
$$\begin{array}{rl}{\mathbb{V}}_{\theta }[s(X;\theta )]& =E_{\theta }[s(X;\theta {)}^2]-E_{\theta }[s(X;\theta ){]}^2\\ & =E_{\theta }[s(X;\theta {)}^2]\\ & ={\int }_x\frac{\partial logf(x;\theta )}{\partial \theta }\frac{\partial logf(x;\theta )}{\partial \theta }f(x;\theta )dx\\ {\int }_x\frac{{\partial }^{2}logf(x;\theta )}{\partial {\theta }^2}f(x;\theta )dx& ={\int }_x\frac{\partial }{\partial \theta }(\frac{1}{f(x;\theta )}\frac{\partial f(x;\theta )}{\partial \theta })dx\\ & ={\int }_x-\frac{(\frac{\partial f(x;\theta )}{\partial \theta }{)}^2}{f(x;\theta {)}^2}+\frac{(\frac{{\partial }^{2}f(x;\theta )}{\partial {\theta }^2})}{f(x;\theta )}f(x;\theta )dx\\ & ={\int }_x-\frac{(\frac{\partial f(x;\theta )}{\partial \theta }{)}^2}{f(x;\theta {)}^2}dx\\ & =-{\int }_x\frac{{\partial }^{2}logf(x;\theta )}{\partial {\theta }^2}f(x;\theta )dx\end{array}$$

渐进正态性

极大似然估计具有渐进正态性
$$\frac{{\hat{\theta }}_n-\theta }{se}\to N(0,1)$$
其中，$se\approx \sqrt{\frac{1}{I_n(\theta )}}\approx \sqrt{\frac{1}{I_n(\hat{\theta })}}$
证明从略，资料比较多。

由此可以构建估计的置信区间。

Bernoulli分布的最大似然估计及其方差

设$X_1,\cdots ,X_n\sim Bernoulli(p)$,则其似然函数是$$L(p)=\prod _{i=1}^n{p}^{X_i}(1-p{)}^{1-X_i}$$$$logL(p)=\sum _i^n{X}_{i}logp+(1-X_i)log(1-p)$$
最大化对数似然，就得到：
$$\begin{array}{rl}& \frac{d}{dx}logL(p)=0\\ & \sum _i^n{X}_i\frac{1}{p}-(1-X_i)\frac{1}{1-p}=0\\ & p=\frac{1}{n}\sum _{i=1}^n{X}_i\end{array}$$
其记分函数是：
$$\frac{\partial logL(p)}{\partial p}=\frac{X}{p}-\frac{1-X}{1-p}$$
$$\begin{gathered} I(p)=-E_{\theta }[\frac{d(\frac{X}{p}-\frac{1-X}{1-p})}{dp}]=\frac{1}{1-p}+\frac{1}{p} \\ =\frac{1}{p(1-p)} \end{gathered}$$
$I_n(p)=nI(p)$，估计的方差$$V(p)=np(1-p)\approx n\hat{p}(1-\hat{p})$$