B站白板推导系列笔记——高斯分布——极大似然估计——有偏or无偏

首先上大佬视频链接：本篇传送门

高斯分布

• 一维高斯分布：

  $$\begin{array}{rl}X& \sim N(\mu ,{\sigma }^2)\\ f(x)& =\frac{1}{\sqrt{2\pi }\sigma }exp(-\frac{(x-\mu {)}^2}{2{\sigma }^2})\end{array}\text{\hspace{1em}(1)}$$

  其中：$\mu$ 是期望，${\sigma }^2$ 是方差

• 高维高斯分布：
  $$\begin{array}{rl}X& \sim N(\mathbf{\mu },\mathbf{\Sigma })\\ f(\mathbf{x})& =\frac{1}{(2\pi {)}^{\frac{d}{2}}|\mathbf{\Sigma }{|}^{\frac{1}{2}}}exp(-\frac{1}{2}(\mathbf{x}-\mathbf{\mu }{)}^T{\mathbf{\Sigma }}^{-1}(\mathbf{x}-\mathbf{\mu }))\end{array}\text{\hspace{1em}(2)}$$

  其中 $\mathbf{x}$ 是 $d$ 维向量，$\mathbf{\mu }$ 是期望, $\mathbf{\Sigma }$ 是协方差矩阵

对高斯分布的参数估计的有偏性

以一维高斯分布为例:
由MLE(最大似然估计)可知:
$${\mu }_{MLE}=\frac{1}{N}\sum _{i=1}^N{x}_i\text{\hspace{1em}(3)}$$

$${\sigma }_{MLE}^2=\frac{1}{N}\sum _{i=1}^N(x_i-{\mu }_{MLE}{)}^2\text{\hspace{1em}(4)}$$

• 讨论 ${\mu }_{MLE}$ 的有偏性:
  $$\begin{array}{r}E[{\mu }_{MLE}]=E[\frac{1}{N}\sum _{i=1}^N{x}_i]=\frac{1}{N}\sum _{i=1}^{N}E[x_i]=\frac{1}{N}\sum _{i=1}^N\mu =\mu \end{array}$$
  故用MLE对 $\mu$ 的估计是无偏的

• 讨论 ${\sigma }_{MLE}^2$ 的有偏性:
  $$\begin{array}{rl}E[{\sigma }_{MLE}^2]& =E[\frac{1}{N}\sum _{i=1}^N(x_i-{\mu }_{MLE}{)}^2]\\ & =\frac{1}{N}\sum _{i=1}^{N}E[(x_i-{\mu }_{MLE}{)}^2]\\ & =\frac{1}{N}\sum _{i=1}^N(E[x_i^2]-E[2{\mu }_{MLE}x]+E[{\mu }_{MLE}^2])\\ & =\frac{1}{N}\sum _{i=1}^N(E[x_i^2]-{\mu }_{MLE}^2)\\ & =\frac{1}{N}\sum _{i=1}^N(E[x_i^2]-{\mu }^2+{\mu }^2-{\mu }_{MLE}^2)\\ & =\frac{1}{N}\sum _{i=1}^N(E[x_i^2]-{\mu }^2)+\frac{1}{N}\sum _{i=1}^N({\mu }^2-{\mu }_{MLE}^2)\\ & =\frac{1}{N}\sum _{i=1}^N(E[x_i^2]-E(x_i{)}^2)+\frac{1}{N}\sum _{i=1}^N({\mu }^2-{\mu }_{MLE}^2)\\ & =\frac{1}{N}\sum _{i=1}^{N}Var(x_i)+\frac{1}{N}\sum _{i=1}^N({\mu }^2-{\mu }_{MLE}^2)\\ & =Var(x_i)+\frac{1}{N}\sum _{i=1}^N({\mu }^2-{\mu }_{MLE}^2)\\ & =Var(x_i)-\frac{1}{N}\sum _{i=1}^N({\mu }_{MLE}^2-{\mu }^2)\\ & =Var(x_i)-\frac{1}{N}\sum _{i=1}^N(E[{\mu }_{MLE}^2]-E[\mu {]}^2)\\ & =Var(x_i)-\frac{1}{N}\sum _{i=1}^N(E[{\mu }_{MLE}^2]-E[{\mu }_{MLE}{]}^2)\\ & =Var(x_i)-\frac{1}{N}\sum _{i=1}^{N}Var({\mu }_{MLE})\\ & =Var(x_i)-Var({\mu }_{MLE})\\ & =Var(x_i)-Var(\frac{1}{N}\sum _{i=1}^N{x}_i)\\ & =Var(x_i)-\frac{1}{N^2}\sum _{i=1}^{N}Var(x_i)\\ & =Var(x_i)-\frac{1}{N}Var(x_i)\\ & =\frac{N-1}{N}{\sigma }^2\end{array}$$
  可知, ${\sigma }_{MLE}^2$ 是有偏的

总结

$E[{\mu }_{MLE}]=\mu ,E[{\sigma }_{MLE}^2]=\frac{N-1}{N}{\sigma }^2$, 即对期望的估计无偏,对方差的估计有偏,故若想得到无偏的方差