机器学习概率基础-高斯分布相关重要知识推导

高斯分布的最大似然估计

数据 $X={\left(x_1,\cdots ,x_N\right)}^{\top }={\left(\begin{array}{c}{x}_1^{\top }\\ ⋮\\ x_N^{\top }\end{array}\right)}_{N\times p},x_i\in {\mathbb{R}}^P,x_i\stackrel{\text{ iid }}{\sim }N(\mu ,\Sigma )$

极大似然估计：${\theta }_{MLE}=argma{x}_{\theta }P(x|\theta )$

高斯分布：
$p(x)=\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)$ 一元高斯
$P(x)=\frac{1}{(2\pi {)}^{\frac{n}{2}}|\Sigma {|}^{\frac{1}{2}}}\exp \left(-\frac{1}{2}(x-\mu {)}^{\top }{\Sigma }^{-1}(x-\mu )\right)$ 多元高斯

一元高斯为例

$\begin{array}{rl}\log P(x|\theta )& =\log {\Pi }_{i=1}^{N}P\left(x_i|\theta \right)={\Sigma }_{i=1}^N\log P\left(x_i|\theta \right)\\ & ={\Sigma }_{i=1}^N\log \frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)\\ & ={\Sigma }_{i=1}^N\left[\log \frac{1}{\sqrt{2\pi }}+\log \frac{1}{\sigma }-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right]\end{array}$

估计均值：

$\begin{array}{rl}{\mu }_{MLE}& =\arg \underset{\mu }{\max }\log P(x|\theta )\\ & =\arg \underset{\mu }{\max }{\Sigma }_{i=1}^N-\frac{(x-\mu {)}^2}{2{\sigma }^2}\\ & =\arg \max {\Sigma }_{i=1}^N{\left(x_i-\mu \right)}^2\end{array}$
$\frac{\partial }{\partial \mu }\Sigma {\left(x_i-\mu \right)}^2={\Sigma }_{i=1}^{N}2\left(x_i-\mu \right)(-1)=0$
$\Rightarrow {\Sigma }_{i=1}^N\left(x_i-\mu \right)=0\Rightarrow {\Sigma }_{i=1}^N{x}_i=N\mu \Rightarrow {\mu }_{MLE}=\frac{1}{N}{\Sigma }_{i=1}^N{x}_i$
$E\left[{\mu }_{MLE}\right]=\frac{1}{N}{\Sigma }_{i=1}^{N}E\left[x_i\right]=\frac{1}{N}{\Sigma }_{i=1}^N\mu =\mu$ (无偏估计)

估计方差：

$\begin{array}{rl}{\sigma }_{MLE}^2& =\arg \max \log P(x|\theta )\\ & =\arg \underset{\sigma }{\max }\left(-\log \sigma -\frac{1}{2{\sigma }^2}{\left(x_i-\mu \right)}^2\right)\end{array}$
令 $L(\sigma )=\left(-\log \sigma -\frac{1}{2{\sigma }^2}{\left(x_i-\mu \right)}^2\right)$
$\frac{\partial L}{\partial \sigma }={\Sigma }_{i=1}^N\left[-\frac{1}{\sigma }+\frac{1}{2}\left(x_i-\mu \right)\cdot 2\cdot {\sigma }^{-3}\right]=0$
$\begin{array}{l}\Rightarrow {\Sigma }_{\overline{i}=1}^N\left[-\frac{1}{\sigma }+\left(x_i-\mu \right){\sigma }^{-3}\right]=0\\ \Rightarrow {\Sigma }_{i=1}^N\left[-{\sigma }^2+\left(x_i-\mu \right)\right]=0\\ \Rightarrow {\Sigma }_{i=1}^N\left(-{\sigma }^2\right)+{\Sigma }_{i=1}^N{\left(x_i-\mu \right)}^2=0\\ \Rightarrow N{\sigma }^2={\Sigma }_{i=1}^N{\left(x_i-\mu \right)}^2\end{array}$
${\sigma }_{MLE}^2=\frac{1}{N}{\Sigma }_{i=1}^N{\left(x_i-{\mu }_{MLE}\right)}^2$(有偏估计)

参数均值与方差的有偏无偏估计

$\theta =\left(\mu ,{\sigma }^2\right),x_i\sim N\left(\mu ,{\sigma }^2\right)$
${\mu }_{MLE}=\frac{1}{N}{\Sigma }_{i=1}^N{x}_i$ (无偏估计)
${\sigma }_{MLE}^2=\frac{1}{N}{\Sigma }_{i=1}^N{\left(x_i-{\mu }_{MLE}\right)}^2$ (有偏估计)

关于均值的无偏估计

$E\left[{\mu }_{MLE}\right]=E\left[\frac{1}{N}{\Sigma }_{i=1}^N{x}_i\right]=\frac{1}{N}{\Sigma }_{i=1}^{N}E\left(x_i\right)=\frac{1}{N}\cdot N\cdot \mu =\mu$

关于方差的有偏估计

${\sigma }_{MLE}^2=\frac{1}{N}{\Sigma }_{i=1}^N{\left(x_i-{\mu }_{MLE}\right)}^2=\frac{1}{N}{\Sigma }_{i=1}^N\left(x_i^2-2\cdot x_i\cdot {\mu }_{MLE}+{\mu }_{MLE}^2\right)$
$=\frac{1}{N}{\Sigma }_{i=1}^N{x}_i^2-\frac{1}{N}{\Sigma }_{i=1}^{N}2\cdot x_i{\mu }_{MLE}+\frac{1}{N}{\Sigma }_{i=1}^N{\mu }_{MLE}^2$
$=\frac{1}{N}{\Sigma }_{i=1}^N{x}_i^2-2\cdot {\mu }_{MLE}^2+{\mu }_{MLE}^2$
$=\frac{1}{N}{\Sigma }_{i=1}^N{x}_i^2-{\mu }_{MLE}^2$

$E\left[{\sigma }_{MLE}^2\right]=E\left[\frac{1}{N}{\Sigma }_{i=1}^N{x}_i^2-{\mu }_{MLE}^2\right]=E\left[\left(\frac{1}{N}{\Sigma }_{i=1}^N{x}_i^2-{\mu }^2\right)-\left({\mu }_{MLE}^2-{\mu }^2\right)\right]$
$=E\left[\frac{1}{N}{\Sigma }_{i=1}^N{x}_i^2-{\mu }^2\right]-E\left[{\mu }_{MLE}^2-{\mu }^2\right]$

$E\left(\frac{1}{N}{\Sigma }_{i=1}^N{x}_i^2-{\mu }^2\right]=E\left[\frac{1}{N}{\Sigma }_{i=1}^N\left(x_i^2-{\mu }^2\right)\right]$
$=\frac{1}{N}{\Sigma }_{i=1}^{N}E\left(x_i^2-{\mu }^2\right)$
$={\sigma }^2$

$E\left[{\mu }_{MLE}^2-{\mu }^2\right]=E\left[{\mu }_{MLE}^2\right]-E\left[{\mu }^2\right]$
$\begin{array}{l}=E\left[{\mu }_{MLE}^2\right]-{\mu }^2\\ =E\left[{\mu }_{MLE}^2\right]-E^2\left[{\mu }_{MLE}\right]\end{array}$
$=Var({\mu }_{MLE})$
$=\mathrm{var}\left(\frac{1}{N}{\Sigma }_{i=1}^N{x}_i\right)$
$\begin{array}{l}=\frac{1}{N^2}{\Sigma }_{i=1}^N\mathrm{Var}\left(x_i\right)=\frac{1}{N^2}{\Sigma }_{i=1}^N{\sigma }^2\\ =\frac{1}{N^2}\cdot N\cdot {\sigma }^2=\frac{1}{N}{\sigma }^2\end{array}$

最终得到

$E\left[{\sigma }_{MLE}^2\right]={\sigma }^2-\frac{1}{N}{\sigma }^2=\frac{N-1}{N}{\sigma }^2̸={\sigma }^2$ (有偏估计)

容易得出利用最大似然估计法估计高斯分布的方差会偏小！！！
极大似然估计针对于高斯分布的方差估计会造成一定的偏差。

实际上其方差的无偏估计为：

${\hat{\sigma }}^2=\frac{1}{N-1}{\Sigma }_{i=1}^N\left(x_i-{\mu }_{MLE}^2\right)$

二维高斯分布的可视化探究

多维高斯分布： $x\sim p(x)=\frac{1}{(2\pi {)}^{\frac{n}{2}}|\Sigma {|}^{\frac{1}{2}}}\exp \left(-\frac{1}{2}(x-\mu {)}^{\top }{\Sigma }^{-1}(x-\mu )\right)$
$x\in {\mathbb{R}}^p,$ p 维的随机向量。
$x=\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_p\end{array}\right),$ $\mu =\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\\ ⋮\\ {\mu }_p\end{array}\right)$, $\Sigma =\left(\begin{array}{cccc}{\sigma }_{11}& {\sigma }_{12}& \dots & {\sigma }_{1p}\\ {\sigma }_{21}& {\sigma }_{22}& \dots & {\sigma }_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ {\sigma }_{P1}& {\sigma }_{p2}& \cdots & {\sigma }_{pp}\end{array}\right)$, 假设$\Sigma$为正定的（一般情况下为半正定）

$(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )$(二次型)， 马氏距离
$\Sigma =1$， 马氏距离为欧氏距离。

$\Sigma$的相似对角化（特征分解）

$\Sigma =u\wedge u^{\top },u{u}^{\top }=u^{\top }u=I$(正交)
$\wedge =\mathrm{diag}\left({\lambda }_i\right),i=1,\cdots ,p_{\cdot },{\lambda }_i\in C$
$U={\left(u_1,\cdots ,u_p\right)}_{p\times p}$

$\Sigma =u\wedge u^{\top }=\left(u_1,\dots u_p\right)\left(\begin{array}{ccc}{\lambda }_1& & \\ & \ddots & \\ & & {\lambda }_p\end{array}\right)\left(\begin{array}{c}{u}_1^{\top }\\ ⋮\\ u_p^{\top }\end{array}\right)$
$=\left(u_1{\lambda }_1\cdots u_p{\lambda }_p\right)\left(\begin{array}{l}{u}_1^{\top }\\ ⋮\\ u_p^{\top }\end{array}\right)$$={\Sigma }_{i=1}^p{u}_i{\lambda }_i{u}_i^{\top }$

${\Sigma }^{-1}={\left(u\wedge u^{\top }\right)}^{-1}={\left(u^T\right)}^{-1}{\wedge }^{-1}{u}^{-1}=u{\Lambda }^{-1}{u}^T={\Sigma }_{i=1}^p{u}_i\frac{1}{\lambda i}{u}_i^T$
${\wedge }^{-1}=\mathrm{diag}\left(\frac{1}{{\lambda }_i}\right),i=1,\cdots ,p$

$\begin{array}{rl}\Delta =(x-\mu ){\Sigma }^{-1}|x-\mu )& =(x-\mu {)}^{\top }{\Sigma }_{i=1}^p{u}_i\frac{1}{\lambda i}\cdot u_i^{\top }(x-\mu )\\ & ={\Sigma }_{i=1}^p(x-\mu {)}^{\top }{u}_1\frac{1}{{\lambda }_i}{u}_i^{\top }(x-\mu )\\ & 令y=\left(\begin{array}{l}{y}_1\\ ⋮\\ y_p\end{array}\right)=(x-\mu {)}^{\top }{u}_i\\ & ={\Sigma }_{i=1}^p{y}_i\frac{1}{{\lambda }_i}{y}_i^T={\Sigma }_{i=1}^p\frac{y_i^2}{{\lambda }_i}\end{array}$

令 p = 2, $\Delta =\frac{y_1^2}{{\lambda }_1}+\frac{y_2^2}{{\lambda }_2}=1({\lambda }_1>{\lambda }_2)$

总结：

令 $\Delta =(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )$
$p(x)=\frac{1}{(2\pi {)}^{\frac{n}{2}}|\Sigma {|}^{\frac{1}{2}}}\exp \left(-\frac{1}{2}\Delta \right)$
令 $p=0.5\to \Delta =r_1$, 对应一个 $\frac{y_1^2}{{\lambda }_1}+\frac{y_2^2}{{\lambda }_2}=r_1$ 椭圆。
令 $p=0.6\to \Delta =r_2$, 对应一个 $\frac{y_1^2}{{\lambda }_1}+\frac{y_2^2}{{\lambda }_2}=r_2$ 椭圆。

当 $\mu =[0,0{]}^T,\Sigma =\left[\begin{array}{cc}1& 0.5\\ 0.5& 1\end{array}\right]$ 如图所示。

高斯分布的局限性

局限一： 参数过大时，难以快速学习

${\Sigma }_{P\times P}\to \frac{P^2-P}{2}+p=\frac{p^2+P}{2}$ 参数个数 $O(p^2)$参数过大 学习困难

1. 假设 ${\Sigma }_{p\times p}$方差矩阵为对角矩阵， $\left(\begin{array}{ccc}{\lambda }_1& & \\ & \ddots & \\ & & {\lambda }_p\end{array}\right)$ 来缩小参数。

2. 假设 ${\Sigma }_{p\times p}$方差矩阵为对角矩阵，且 ${\lambda }_1={\lambda }_2=...={\lambda }_p=\lambda$ 即就是$\left(\begin{array}{ccc}{\lambda }_{}& & \\ & \ddots & \\ & & {\lambda }_{}\end{array}\right)$ 来进一步缩小参数。

factor analysis 假设为 对角矩阵。
P-PCA 各向同性。

局限二： 有些时候用一个高斯分布建模难以确切表达模型。

GMM （高斯混合模型）采用多个高斯分布建模。

求多维高斯分布的边缘分布和条件概率分布

已知高维高斯分布的随机变量，均值，协方差:
  ~  
$x=\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_p\end{array}\right)\mu =\left(\begin{array}{c}{\mu }_1\\ {\mu }_2\\ ⋮\\ {\mu }_p\end{array}\right)\Sigma ={\left(\begin{array}{cccc}{\sigma }_{11}& {\sigma }_{12}& \cdots & {\sigma }_{1p}\\ {\sigma }_{21}& {\sigma }_{22}& \cdots & {\sigma }_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ {\sigma }_{p1}& {\sigma }_{p2}& \cdots & {\sigma }_{pp}\end{array}\right)}_{p\times p}$
  ~  
由于要求条件概率以及边缘概率，将随机变量分成两组：
  ~  
$x=\left(\begin{array}{l}{x}_a\\ x_b\end{array}\right)\begin{array}{l}(m)\\ (n)\end{array}m+n=p\mu =\left(\begin{array}{l}{\mu }_a\\ {\mu }_b\end{array}\right)\Sigma =\left(\begin{array}{ll}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ba}& {\Sigma }_{bb}\end{array}\right)$
  ~  
求：$P\left(x_a\right),P\left(x_b|x_a\right),P\left(x_b\right),P\left(x_a|x_b\right)$

在 PRML 中通用方法是配方法,思想简单，但计算量大。这里不使用。

引入定理： $x\sim N(\mu ,\Sigma ),y=Ax+B,$ 则 $y\sim N\left(A\mu +B,A\Sigma A^{\top }\right)$
  ~  
$E[y]=E[Ax+B]=AE[x]+B=A\mu +B$
$\mathrm{var}[y]=\mathrm{var}[Ax+B]=\mathrm{var}[Ax]+\mathrm{var}(B]=A\mathrm{var}[x]A^T=A\Sigma A^T$

解：
构造：$x_a=Ax=\left(I_{m}0\right)\left(\begin{array}{l}{x}_a\\ x_b\end{array}\right)=x_a$
有上述定理得:
$E\left[x_a\right]=\left(I_{m}0\right)\left(\begin{array}{l}{\mu }_a\\ {\mu }_b\end{array}\right)={\mu }_a$
$\mathrm{var}[y]=\mathrm{var}[Ax+B]=\mathrm{var}[Ax]+\mathrm{var}[B]=A\mathrm{Var}[x]A^T=A\Sigma A^T$
所以：$x_a\sim N\left({\mu }_a,{\Sigma }_{aa}\right)$
  ~  
求$x_b|x_a$
    ~~~    
定义三个变量：
$\begin{array}{l}{x}_{b\cdot a}=x_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a\\ {\mu }_{b\cdot a}={\mu }_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\mu }_a\\ {\Sigma }_{bb\cdot a}={\Sigma }_{bb}-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\Sigma }_{ab}\end{array}$
  ~  
   ~~   其中${\Sigma }_{bb\cdot a}$ 称为 schur complementary. (线性代数)
  ~  
那么易构造$x_{b\cdot a}=\left(-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{I}_n\right)\left(\begin{array}{l}{x}_a\\ x_b\end{array}\right)$
由上述定理得：
$E\left[x_{b\cdot a}\right]=\left(-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{I}_n\right)\left(\begin{array}{l}{\mu }_a\\ {\mu }_b\end{array}\right)={\mu }_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\mu }_a={\mu }_{b\cdot a}$
$\mathrm{var}\left[x_{b\cdot a}\right]=\left(-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{I}_n\right)\left(\begin{array}{l}{\Sigma }_{aa}^{-1}{\Sigma }_{ab}\\ {\Sigma }_{ba}{\Sigma }_{bb}\end{array}\right)\left(\begin{array}{c}-{\Sigma }_{aa}^{-1}{\Sigma }_{ba}^T\\ I_n\end{array}\right)$
$==\left(0\cdot {\Sigma }_{bb}-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\Sigma }_{ab}\right)\left(\begin{array}{c}-{\Sigma }_{aa}^{-1}{\Sigma }_{ba}^T\\ I_n\end{array}\right)={\Sigma }_{bb}-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\Sigma }_{ab}={\Sigma }_{bb\cdot a}$
$x_{b\cdot a}\sim N\left({\mu }_{b\cdot a},{\Sigma }_{bb\cdot a}\right)$
$x_{b\cdot a}=x_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a\Rightarrow x_b=x_{ba}+{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a$ 这里 $x_a$为常数。
  ~  
$\begin{array}{l}E\left[x_b|x_a\right]={\mu }_{b\cdot a}+{\sum }_{ba}{\sum }_{aa}^{-1}{x}_a\\ \mathrm{var}\left[x_b|x_a\right]=\mathrm{var}\left[x_{b\cdot a}\right]={\sum }_{bb\cdot a}\end{array}$
  ~  
$x_b|x_a\sim N\left({\mu }_{ba}+{\Sigma }_{b\cdot a}{\Sigma }_{aa}^{-1}{x}_a,{\Sigma }_{bb\cdot a}\right)$