白板推导2-高斯分布

1:用极大似然求高斯分布参数

令$data:X=(x_1{x}_2...x_p{)}^T$
$x_i\in R^p$
$x_i服从N(\mu ,{\sigma }^2)$ ,是独立同分布
一维高斯概率密度函数：
p ( x ) = 1 2 π σ exp ⁡ { − ( x − μ ) 2 2 σ 2 } p(x)=\frac{1}{\sqrt{2\pi}\sigma}\exp\{-\frac{(x-\mu)^2}{2\sigma^2}\} p(x)=2π ​σ1​exp{−2σ2(x−μ)2​}
多维高斯概率密度函数：
p ( x ) = 1 ( 2 π ) p 2 ∣ Σ ∣ 1 2 exp ⁡ { − 1 2 ( x − μ ) T Σ − 1 ( x − μ ) } p(x) =\frac{1}{(2\pi)^{\frac{p}{2}}|\Sigma|^{\frac{1}{2}}}\exp\{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)\} p(x)=(2π)2p​∣Σ∣21​1​exp{−21​(x−μ)TΣ−1(x−μ)}
极大似然法求一维高斯的参数
$\theta =(\mu ,{\sigma }^2)$
对原概率函数求对数，对数函数和原函数有相同的单调性
$\log p(x|\theta )=\log \prod _{i=1}^{n}p(x_i|\theta )$
$=\sum _{i=1}^n\log p(x_i|\theta )$
= ∑ i = 1 n log ⁡ 1 2 π σ exp ⁡ { − ( x i − μ ) 2 2 σ 2 } = \displaystyle \sum_{i=1}^n \log \frac{1}{\sqrt{2\pi}\sigma}\exp\{-\frac{(x_i-\mu)^2}{2\sigma^2}\} =i=1∑n​log2π ​σ1​exp{−2σ2(xi​−μ)2​}
$=\sum _{i=1}^n(\log \frac{1}{\sqrt{2\pi }}+\log \frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2})$

$ML{E}_{\mu }=argma{x}_{\mu }\log p(x|\theta )$
$=argma{x}_{\mu }\sum _{i=1}^n(\log \frac{1}{\sqrt{2\pi }}+\log \frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2})$
对$\mu 求偏导并令其为0有$
$\sum _{i=1}^{n}2(x_i-\mu )(-1)=0$
$\sum _{i=1}^n{x}_i=\sum _{i=1}^n\mu$
$\mu =\frac{1}{n}\sum _{i=1}^n{x}_i$

对${\sigma }^2求偏导并令其为0有$
$\sum _{i=1}^n(-\frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2}\ast (-2){\sigma }^{-3})=0$
$\sum _{i=1}^n((x_i-\mu {)}^2-{\sigma }^2)=0$
${\sigma }^2=\frac{1}{n}\sum _{i=1}^n(x_i-\mu {)}^2=0$

2：高斯分布的几何解释

多维高斯分布公式：
p ( x ) = 1 ( 2 π ) p 2 ∣ Σ ∣ 1 2 exp ⁡ { − 1 2 ( x − μ ) T Σ − 1 ( x − μ ) } p(x) =\displaystyle \frac{1}{(2\pi)^{\frac{p}{2}}|\Sigma|^{\frac{1}{2}}}\exp\{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)\} p(x)=(2π)2p​∣Σ∣21​1​exp{−21​(x−μ)TΣ−1(x−μ)}
其中
$x_i\in R^p$
$$X=\left[\begin{array}{c}{x}_1\\ x_2\\ ...\\ x_n\end{array}\right]\mu =\left[\begin{array}{c}{\mu }_1\\ {\mu }_2\\ ...\\ {\mu }_n\end{array}\right]\Sigma =\left[\begin{array}{c}{\sigma }_{11}{\sigma }_{12}...{\sigma }_{1p}\\ {\sigma }_{21}{\sigma }_{22}...{\sigma }_{2p}\\ .........\\ {\sigma }_{n1}{\sigma }_{n2}...{\sigma }_{np}\end{array}\right]并假定\Sigma 是正定的$$
多维高斯分布只有指数项部分和x相关，所以可以只研究指数项部分，其中
$(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )称为x与\mu 的马氏距离$
首先对协方差矩阵做特征值分解,因为是正定矩阵有：

$\Sigma =U\lambda U^T,U{U}^T=U^{T}U=U^{-1}U=I,\lambda =diag({\lambda }_i)$
${\Sigma }^{-1}=(U\lambda U^T{)}^{-1}=(U^T{)}^{-1}{\lambda }^{-1}{U}^{-1}=U{\lambda }^{-1}{U}^T$
$$\Sigma =U\lambda U^T=(u_1{u}_2...u_p)\left[\begin{array}{c}{\lambda }_1..............0\\ 0...{\lambda }_2........0\\ 0.........{\lambda }_i....0\\ 00...........{\lambda }_p\end{array}\right](u_1{u}_2...u_p{)}^T$$
$$\begin{gathered} =(u_1{\lambda }_1{u}_2{\lambda }_2...u_p{\lambda }_p)(u_1{u}_2...u_p{)}^T \\ =\sum u_i{\lambda }_i{u}_i^T \end{gathered}$$
所以有
${\Sigma }^{-1}=\sum _{i=1}^p{u}_i\frac{1}{{\lambda }_i}{u}_i^T$
带入马氏距离有：
$$\begin{gathered} (x-\mu {)}^T{\Sigma }^{-1}(x-\mu )=(x-\mu {)}^T\sum _{i=1}^p{u}_i\frac{1}{{\lambda }_i}{u}_i^T(x-\mu ) \\ 把求和符号提到前面有 \\ =\sum _{i=1}^p(x-\mu {)}^T{u}_i\frac{1}{{\lambda }_i}{u}_i^T(x-\mu ) \end{gathered}$$
令$y_i=(x-\mu {)}^T{u}_i$
原式=$\sum _{i=1}^p\frac{y_i^2}{{\lambda }_i}$
令p=2,也就是3维的情况，有
$p(x)=\frac{y_1^2}{{\lambda }_1}+\frac{y_2^2}{{\lambda }_2}$
当概率给定时，比如r,有$p(x)=\frac{y_1^2}{{\lambda }_1}+\frac{y_2^2}{{\lambda }_2}=r$,从几何角度看，该图形是以$\lambda 为轴，u方向的椭圆$

概率从0到1的整个形状，如下图所示，像山峰一样

3：高斯分布的局限性

1. 协方差参数过多，假定$x\in R^p,协方差参数为p^2,为了简化操作有时可以假定协方差矩阵为对角矩阵$
2. 当个的高斯分布不一定符合实际问题，有时可以采用混合高斯模型