高斯分布基础知识详解

目录

高斯分布

linear Gaussian model

$z_{t}=Az_{t-1}+B+\epsilon$其中$\epsilon$为噪音

极大似然估计

• 已知
  $X=({\mathrm{x}}_1,{\mathrm{x}}_2,{\mathrm{x}}_3,\cdots {\mathrm{x}}_N{)}^T$，$X_i\in \mathbb{R}$
• 推导$X_i\sim \mathbb{N}(\mu ,{\sigma }^2)$，$X_i\in {\mathbb{R}}^P$，

$log\mathrm{P}(X|\theta )=log{\prod }_{i=1}^N\mathrm{P}(X|\theta )={\sum }_{i=1}^{N}log\mathrm{P}(x_i|\theta )={\sum }_{i=1}^N[\log \frac{1}{\sqrt{2\pi }}+\log \frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}]$

${\mu }_{MLE}=\underset{\mu }{\mathrm{argmax}}\log \mathrm{P}(X|\theta )=\underset{\mu }{\mathrm{argmin}}{\sum }_{i=1}^N(x_i-\mu {)}^2$

得到${\mu }_{MLE}=\frac{1}{N}{\sum }_{i=1}^N{x}_i$

同理${\sigma }_{MLE}^2=\frac{1}{N}{\sum }_{i=1}^N(x_i-{\mu }_{MLE}{)}^2$

• ${\mu }_{MLE}$叫无偏估计
• ${\sigma }_{MLE}^2$叫有偏估计（$E[{\sigma }_{MLE}^2]=\frac{N-1}{N}{\sigma }^2$）
• 无偏$\hat{\sigma }=\frac{1}{N-1}{\sum }_{i=1}^N(x_i-\mu {)}^2$
  • 推导 $E[{\sigma }_{MLE}^2]=E[\frac{1}{N}{\sum }_{i=1}^N(x_i^2-2x_i{\mu }_{MLE}+{\mu }_{MLE}^2)]=E[\frac{1}{N}{\sum }_{i=1}^N(x_i^2-2{\mu }_{MLE}^2+{\mu }_{MLE}^2)]=E[\frac{1}{N}{\sum }_{i=1}^N{x}_i^2-{\mu }^2]-E[{\mu }_{MLE}^2-{\mu }^2)]={\sigma }^2-(E[{\mu }_{MLE}^2]-E^2[{\mu }_{MLE}])=\frac{N-1}{N}{\sigma }^2$
• 极大似然估计方差偏小

多维高斯分布

• $x\sim \mathbb{N}(\mu ,\sum )=\frac{1}{(2\pi {)}^{\frac{P}{2}}|\sum {|}^{\frac{1}{2}}}exp(-\frac{1}{2}(x-\mu {)}^T{\sum }^{-1}(x-\mu ))$
  其中$\mu =({\mu }_1,{\mu }_2,\cdots ,{\mu }_P{)}^T$,$\sum =\left(\begin{array}{llc}{\sigma }_{11}& \cdots & {\sigma }_{1P}\\ ⋮& ⋮& ⋮\\ {\sigma }_{P1}& \cdots & {\sigma }_{PP}\end{array}\right)$`（方差矩阵）为正交的且为半正定（一般）

• $(x-\mu {)}^T{\sum }^{-1}(x-\mu )$:马氏距离（x与$\mu$的距离）

  • $\sum =I$,马氏距离=欧氏距离
  • example:
    • $z_1=(z_{11},z_{12}),z_2=(z_{21},z_{22})$,$(z_1-z_2{)}^T{\sum }^{-1}(z_1-z_2)=(z_{11}-z_{21}{)}^2-(z_{12}-z_{22}{)}^2$为欧氏距离
  • $\sum =U\lambda U^T=U^{T}U=1$,其中$U{U}^T=U^{T}U=1$，$\lambda$为特征值矩阵
    • $\sum =U\lambda U^T=U^{T}U=1={\sum }_{i=1}^P{u}_i\lambda u_i^T$
    • ${\sum }^{-1}=(U\lambda U^T{)}^{-1}={\sum }_{i=1}^P{u}_i{\lambda }^{-1}{u}_i^T$
  • $\Delta =(x-\mu {)}^T{\sum }^{-1}(x-\mu )={\sum }_{i=1}^P(x-\mu {)}^T{u}_i{\lambda }^{-1}{u}_i^T(x-\mu )={\sum }_{i=1}^P{y}_i\frac{1}{{\lambda }_i}{y}_i^T={\sum }_{i=1}^P\frac{y_i^2}{{\lambda }_i}$,其中$y_i=(x-\mu {)}^T{\mu }_i$
    • $p=2$,$\Delta =\frac{y_1^2}{{\lambda }_1}+\frac{y_2^2}{{\lambda }_1}=r_i$，便可以将其看作是椭圆，之不过是变换轴,这样就会出现等高线，即可与二维的高斯分布对比。

• ${\sum }_{p\times p}$

  • 有$\frac{p^2-p}{2}+p=\frac{p^2+p}{2}$个参数即有$\frac{p(p+1)}{2}=O(p^2)$

• 局限性：如果样本点拟合成两个高斯分布更准确，但是实际中使用高斯则是用一个大的高斯去拟合，这样就会存在造成较大误差。

已知高斯分布求边缘和条件高斯分布

• 已知$x=\left(\begin{array}{l}{x}_a\\ x_b\end{array}\right)$,$\mu =\left(\begin{array}{l}{\mu }_a\\ {\mu }_b\end{array}\right)$,$\sum =\left(\begin{array}{ll}{\sigma }_{aa}& {\sigma }_{ab}\\ {\sigma }_{ba}& {\sigma }_{bb}\end{array}\right)$,其中$a+b=p$。求$P(x_a)$,$P(x_b|x_a)$,$P(x_b)$,$P(x_a|x_b)$
• 配方法（prml）
• 定理：$x\sim N(\mu ,{\sigma }^2)$,$y=AX+B$;结论：$y\sim N(A\mu +B,A\sum A^T)$
  • $x_a=(I_m,0)\left(\begin{array}{l}{x}_a\\ x_b\end{array}\right)$`
    • `$E(x_a)=E((I_m,0)\left(\begin{array}{l}{x}_a\\ x_b\end{array}\right))={\mu }_a$
    • `$var(x_a)=var((I_m,0)\left(\begin{array}{ll}{\sum }_{aa}& {\sum }_{ab}\\ {\sum }_{ba}& {\sum }_{bb}\end{array}\right))={\sigma }_{aa}$
    • `$x_a\sim N({\mu }_a,{\sum }_{aa})$
  • 求$P(x_a|x_b)$
    • 构造$x_{b-a}=x_b-x_{bb}{\sum }^{-1}{x}_a$,${\mu }_{b-a}={\mu }_b-{\mu }_{bb}{\sum }^{-1}{\mu }_a$,${\sum }_{b-a}={\sum }_{bb}-{\sum }_{ba}{\sum }^{-1}{\sum }_{ab}$`
    • $E(x_b|x_a)=E(x_a)$=

已知边缘概率密度求联合

• 已知$p(x)=N(x|\mu ,{\lambda }^{-1})$,$p(x|y)=N(y|Ax+B,L^{-1})$。求$p(y)$,$p(x|y)$
  • 对于线性高斯$y=AX+B+ϵ,ϵ\sim N(0,L^{-1}),ϵ与x独立$
  • 解$E[y]=E[Ax+B+ϵ]=A\mu +B$,$var[Y]=var[AX+B+ϵ]=var[AX+B]+var[ϵ]=A{\lambda }^{-1}{A}^T+L^{-1}$,得到$y\sim N（A\mu +B,A{\lambda }^{-1}{A}^T+L^{-1}）$
  • $z=\left(\begin{array}{l}x\\ y\end{array}\right)\sim N\left(\left[\begin{array}{l}\mu \\ A\mu +b\end{array}\right],\left[\begin{array}{ll}{\lambda }^{-1}& \Delta \\ \Delta & L^{-1}+A{\lambda }^{-1}{A}^T+L^{-1}\end{array}\right]\right)$，其中$\Delta =cov(x,y)=E[(X-\mu )(y-E[y]{)}^T]=E[(X-\mu )(AX-A\mu +ϵ)]=E[(X-\mu )(X-\mu {)}^T{A}^T]=E[(X-\mu )(X-\mu {)}^T]A^T=Var[x]A^T={\lambda }^{-1}{A}^T$
    -[x] $p(x|y)\sim$