机器学习-白板推导 P2_1

一维高斯分布：

高斯分布

$$\begin{array}{rl}& X:data\to X=(x_1{x}_2...x_N{)}_{N\times p}^T\\ & x_i\in R^p\\ & x_i{\sim }^{iid}N(\mu ,\Sigma )\\ & \theta =(\mu ,\Sigma )\end{array}$$

一维高斯分布：

$$p(x)=\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)$$

p维高斯分布：

$$p(x)=\frac{1}{{2\pi }^{\frac{p}{2}}}\exp \left(-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )\right)$$

参数值公式推导

$${\theta }_{MLE}=arg\underset{\theta }{\max }\log p(x|\theta )$$
令$p=1$,$\theta =(\mu ,{\sigma }^2)$
$$\begin{array}{rl}\log p(x|\theta )& =\log \prod _{i=1}^{N}p(x_i|\theta )\\ & =\sum _{i=1}^N\log p(x_i|\theta )\\ & =\sum _{i=1}^N\log \frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\right)\\ & =\sum _{i=1}^N\left[\log \frac{1}{\sqrt{2\pi }}+\log \frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\right]\end{array}$$
求 $\mu$:
$$\begin{array}{rl}{\mu }_{MLE}& =arg\underset{\mu }{\max }\log p(x|\theta )\\ & =arg\underset{\mu }{\max }\sum _{i=1}^N-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\\ & =arg\underset{\mu }{\min }\sum _{i=1}^N(x_i-\mu {)}^2\end{array}$$
$$\begin{array}{rl}\frac{\partial }{\partial \mu }\sum _{i=1}^N(x_i-\mu {)}^2& =\sum _{i=1}^{N}2\ast (x_i-\mu )\ast (-1)=0\\ \sum _{i=1}^N(x_i-\mu )& =0\\ \sum _{i=1}^N{x}_i-\sum _{i=1}^N\mu & =0\end{array}$$
$${\mu }_{MLE}=\frac{1}{N}\sum _{i=1}^N{x}_i$$

求 ${\sigma }^2$:
$$\begin{array}{rl}{\sigma }_{MLE}^2& =arg\underset{\sigma }{\max }\log p(x|\theta )\\ & =arg\underset{\sigma }{\max }\sum _{i=1}^N\left(\log \frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\right)\end{array}$$
$$\begin{array}{rl}\frac{\partial }{\partial \sigma }\sum _{i=1}^N\left(\log \frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\right)& =\sum _{i=1}^N\left(-\frac{1}{\sigma }+(x_i-\mu {)}^2\ast {\sigma }^{-3}\right)=0\\ \sum _{i=1}^N\left(-{\sigma }^{-2}+(x_i-\mu {)}^2\right)& =0\\ -\sum _{i=1}^N{\sigma }^2+\sum _{i=1}^N(x_i-\mu {)}^2& =0\\ \sum _{i=1}^N{\sigma }^2=\sum _{i=1}^N(x_i-\mu {)}^2& \end{array}$$
$${\sigma }_{MLE}^2=\frac{1}{N}\sum _{i=1}^N(x_i-\mu {)}^2$$

参数有偏无偏推导

${\mu }_{MLE}$为无偏估计
$$E[{\mu }_{MLE}]=\frac{1}{N}\sum _{i=1}^{N}E[x_i]=\frac{1}{N}\sum _{i=1}^N\mu =\mu$$

$$\begin{array}{rl}Var[{\mu }_{MLE}]& =Var[\frac{1}{N}\sum _{i=1}^N{x}_i]=\frac{1}{N^2}\sum _{i=1}^{N}Var[x_i]\\ & =\frac{1}{N^2}\sum _{i=1}^N{\sigma }^2=\frac{1}{N^2}N{\sigma }^2=\frac{1}{N}{\sigma }^2\end{array}$$

${\sigma }_{MLE}^2$ 为有偏估计。
无偏估计值应为：
$$E[{\sigma }_{MLE}^2]=\frac{N-1}{N}{\sigma }^2\hat{\sigma }=\frac{1}{N-1}\sum _{i=1}^N(x_i-\mu {)}^2$$
${\sigma }_{MLE}^2$ 公式推导：
$$\begin{array}{rl}{\sigma }_{MLE}^2& =\frac{1}{N}\sum _{i=1}^N(x_i-{\mu }_{MLE}{)}^2\\ & =\frac{1}{N}\sum _{i=1}^N(x_i^2-2x_i{\mu }_{MLE}+{\mu }_{MLE}^2)\\ & =\frac{1}{N}\sum _{i=1}^N{x}_i^2-\frac{1}{N}\sum _{i=1}^{N}2x_i{\mu }_{MLE}+\frac{1}{N}\sum _{i=1}^N{\mu }_{MLE}^2\\ & =\frac{1}{N}\sum _{i=1}^N{x}_i^2-{\mu }_{MLE}^2\end{array}$$
$$\begin{array}{rl}E[{\sigma }_{MLE}^2]& =E[\frac{1}{N}\sum _{i=1}^N{x}_i^2-{\mu }_{MLE}^2]\\ & =E[(\frac{1}{N}\sum _{i=1}^N{x}_i^2-{\mu }^2)-({\mu }_{MLE}^2-{\mu }^2)]\\ & =E[\frac{1}{N}\sum _{i=1}^N{x}_i^2-{\mu }^2]-E[{\mu }_{MLE}^2-{\mu }^2]\end{array}$$
$$\begin{array}{rl}E[\frac{1}{N}\sum _{i=1}^N{x}_i^2-{\mu }^2]& =E[\frac{1}{N}\sum _{i=1}^N(x_i^2-{\mu }^2)]\\ & =\frac{1}{N}\sum _{i=1}^{N}E[(x_i^2-{\mu }^2)]\\ & =\frac{1}{N}\sum _{i=1}^N(E[x_i^2]-{\mu }^2)\\ & =\frac{1}{N}\sum _{i=1}^N(Var[x_i])\\ & =\frac{1}{N}\sum _{i=1}^N({\sigma }^2)\\ & ={\sigma }^2\end{array}$$
$$\begin{array}{rl}E[{\mu }_{MLE}^2-{\mu }^2]& =E[{\mu }_{MLE}^2]-E[{\mu }^2]\\ & =E[{\mu }_{MLE}^2]-{\mu }^2\\ & =E[{\mu }_{MLE}^2]-{E[{\mu }_{MLE}]}^2\\ & =Var[{\mu }_{MLE}]\\ & =\frac{1}{N}{\sigma }^2\end{array}$$
$$E[{\sigma }_{MLE}^2]={\sigma }^2-\frac{1}{N}{\sigma }^2=\frac{N-1}{N}{\sigma }^2$$

B站链接：
https://www.bilibili.com/video/av32905863?from=search&seid=8309397892501615322