14-高斯分布基础知识

1.高斯分布参数估计/样本数据定义

定义数据样本 D = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . . , ( x N , y N ) } , 其 中 x i ∈ R p , y i ∈ R , 其 中 x i = 1 , 2 , . . . , N D=\{(x_1,y_1),(x_2,y_2),...,(x_N,y_N)\},其中x_i\in \mathbb{R}^p,y_i\in \mathbb{R},其中x_i = 1,2,...,N D={(x1​,y1​),(x2​,y2​),...,(xN​,yN​)},其中xi​∈Rp,yi​∈R,其中xi​=1,2,...,N
$$X=(x_1,x_2,...,x_N{)}^T=\left(\begin{array}{c}{x}_1^T\\ x_2^T\\ ⋮\\ x_N^T\end{array}\right)={\left(\begin{array}{cccc}{x}_{11}& x_{12}& ...& x_{1p}\\ x_{21}& x_{22}& ...& x_{2p}\\ ⋮& ⋮& ⋮& ⋮\\ x_{N1}& x_{N2}& ...& x_{Np}\end{array}\right)}_{N\times P}\text{\hspace{1em}(1)}$$
其中$x_i$是P维列向量，且$x_i\sim N(\mu ,\Sigma )$,我们可以看出$\theta =N(\mu ,\Sigma )$

2.极大似然法求${\mu }_{MLE},{\sigma }_{MLE}^2$

2.1 $L(\theta )表达式$

我们知道每一个单变量$x_i$服从高斯分布，由高斯分布密度函数为：
$$P(x_i|\theta )=\frac{1}{\sqrt{2\pi }\sigma }ex{p}^{(-\frac{(x-\mu {)}^2}{2{\sigma }^2})}\text{\hspace{1em}(2)}$$
我们希望通过观察到的数据来计算参数$\theta$的值，那么我们可以使用极大似然估计法求,
由于我们假设$x_i是服从独立同分布的，所以可得如下：$
$${\theta }_{MLE}=argma{x}_{\theta }\log \prod _{i=1}^{N}p(x_i|\theta )=argma{x}_{\theta }\sum _{i=1}^N\log p(x_i|\theta )\text{\hspace{1em}(3)}$$
代入$P(x_i|\theta )$可得：
$$L(\theta )=argma{x}_{\theta }\sum _{i=1}^N\log [\frac{1}{\sqrt{2\pi }}]-\sum _{i=1}^N\log [\sigma ]-\sum _{i=1}^N\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\text{\hspace{1em}(4)}$$
为方便计算，我们令：①=${\sum }_{i=1}^N\log [\frac{1}{\sqrt{2\pi }}]$;②=${\sum }_{i=1}^N\log [\sigma ]$;③=${\sum }_{i=1}^N\frac{(x_i-\mu {)}^2}{2{\sigma }^2}$
那么$L(\theta )=①-②-③$

2.2 极大似然法求${\mu }_{MLE}$

由极大似然法我们可以得到只需要将$L(\theta )$对$\mu$求偏导，并令其为零。
$$\frac{\partial L(\theta )}{\partial \mu }=0\text{\hspace{1em}(5)}$$
展开上式可得如下：
$$\frac{\partial L(\theta )}{\partial \mu }=0-0-\sum _{i=1}^N\frac{2(x_i-\mu )}{2{\sigma }^2}(-1)=0\text{\hspace{1em}(6)}$$
整理上式可得：
$$\sum _{i=1}^N(x_i-\mu )=0\text{\hspace{1em}(7)}$$
$$\sum _{i=1}^N{x}_i=\sum _{i=1}^N\mu =N\mu \text{\hspace{1em}(8)}$$
$$结论：{\mu }_{MLE}=\frac{{\sum }_{i=1}^N{x}_i}{N}\text{\hspace{1em}(9)}$$

2.3无偏估计${\mu }_{MLE}$

由概率知识可得，如果我们求得$\mathbb{E}[{\mu }_{MLE}]=\mu ;那么{\mu }_{MLE}就是无偏估计；$
$$$\mathbb{E}[{\mu }_{MLE}]=\mathbb{E}[\frac{{\sum }_{i=1}^N{x}_i}{N}]=\frac{1}{N}\mathbb{E}[\sum _{i=1}^N{x}_i]=\frac{1}{N}N\mu =\mu \text{\hspace{1em}(10)}$$

2.4 极大似然法求${\sigma }_{MLE}^2$

由极大似然法我们可以得到只需要将$L(\theta )$对$\sigma$求偏导，并令其为零。
$$\frac{\partial L(\theta )}{\partial \sigma }=0\text{\hspace{1em}(11)}$$
展开上式可得如下：
$$\frac{\partial L(\theta )}{\partial \sigma }=0-\sum _{i=1}^N\frac{1}{\sigma }+\sum _{i=1}^N(x_i-\mu {)}^2{\sigma }^{-3}\text{\hspace{1em}(12)}$$
整理上式可得：
$$结论：{\sigma }_{MLE}^2=\frac{1}{N}\sum _{i=1}^N(x_i-{\mu }_{MLE}{)}^2\text{\hspace{1em}(13)}$$

2.5 有偏估计${\sigma }_{MLE}^2$

由极大似然法可得：
$${\sigma }_{MLE}^2=\frac{1}{N}\sum _{i=1}^N(x_i-{\mu }_{MLE}{)}^2=\frac{1}{N}\sum _{i=1}^N{x}_i^2+\frac{1}{N}\sum _{i=1}^N{\mu }_{MLE}^2-\frac{1}{N}\sum _{i=1}^{N}2x_i{\mu }_{MLE}\text{\hspace{1em}(14)}$$
由于${\mu }_{MLE}=\frac{{\sum }_{i=1}^N{x}_i}{N}$所以可得如下：
$${\sigma }_{MLE}^2=\frac{1}{N}\sum _{i=1}^N{x}_i^2+\frac{N}{N}{\mu }_{MLE}^2-2{\mu }_{MLE}^2=\frac{1}{N}\sum _{i=1}^N{x}_i^2-{\mu }_{MLE}^2\text{\hspace{1em}(15)}$$
由期望方差关系公式可得：
$$\mathbb{E}[X^2]-[\mathbb{E}(X){]}^2=\mathbb{D}(X)\text{\hspace{1em}(16)}$$
$$\mathbb{E}[\frac{1}{N}\sum _{i=1}^N{x}_i^2]=\frac{1}{N}\sum _{i=1}^N\mathbb{E}[x_i^2]=\frac{1}{N}\sum _{i=1}^N([\mathbb{E}{(x_i)]}^2+D(x_i))=\frac{1}{N}\sum _{i=1}^N[{\mu }^2+{\sigma }^2]={\mu }^2+{\sigma }^2\text{\hspace{1em}(17)}$$
$$\mathbb{E}[{\mu }_{MLE}^2]=(\mathbb{E}({\mu }_{MLE}){)}^2+D({\mu }_{MLE})={\mu }^2+D(\frac{{\sum }_{i=1}^N{x}_i}{N})={\mu }^2+\frac{1}{N^2}\sum _{i=1}^{N}D(x_i)={\mu }^2+\frac{1}{N}{\sigma }^2\text{\hspace{1em}(18)}$$
$$\mathbb{E}[{\sigma }_{MLE}^2]={\mu }^2+{\sigma }^2-({\mu }^2+\frac{1}{N}{\sigma }^2)=\frac{N-1}{N}{\sigma }^2\text{\hspace{1em}(19)}$$
$$故：\mathbb{E}[{\sigma }_{MLE}^2]\ne {\sigma }^2\text{\hspace{1em}(20)}$$
$$结论：{\sigma }_{MLE}^2是有偏估计$$
注：我们在求${\sigma }_{MLE}^2时，我们用的是{\mu }_{MLE}，而不是\mu ,而{\mu }_{MLE}是我们通过样本数据来生成的$，那么它就一定会出现部分误差，并且用局部样本去估算整体样本时会偏小，无法完全覆盖整个样本。所以，我们在高斯正太分布中，利用极大似然估计求出来的${\sigma }_{MLE}^2$偏小。

3.多变量的高斯分布

3.1 多变量的高斯分布样本定义：

多变量的高斯分布$X\sim N(\mu ,\Sigma )$,其概率密度函数如下：
$$P(x_i|\theta )=\frac{1}{(2\pi {)}^{\frac{p}{2}}|\Sigma {|}^{\frac{1}{2}}}ex{p}^{(-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))}\text{\hspace{1em}(21)}$$
$$X=(x_1,x_2,...,x_N{)}^T={\left(\begin{array}{c}{x}_1^T\\ x_2^T\\ ⋮\\ x_N^T\end{array}\right)}_{P\times 1}={\left(\begin{array}{cccc}{x}_{11}& x_{12}& ...& x_{1p}\\ x_{21}& x_{22}& ...& x_{2p}\\ ⋮& ⋮& ⋮& ⋮\\ x_{N1}& x_{N2}& ...& x_{Np}\end{array}\right)}_{N\times P}\text{\hspace{1em}(22)}$$
$$\Sigma ={\left(\begin{array}{cccc}{\sigma }_{11}& {\sigma }_{12}& ...& {\sigma }_{1p}\\ {\sigma }_{21}& {\sigma }_{22}& ...& {\sigma }_{2p}\\ ⋮& ⋮& ⋮& ⋮\\ {\sigma }_{p1}& {\sigma }_{p2}& ...& {\sigma }_{pp}\end{array}\right)}_{p\times P};\mu ={\left(\begin{array}{c}{\mu }_1^T\\ {\mu }_2^T\\ ⋮\\ {\mu }_p^T\end{array}\right)}_{P\times 1}\text{\hspace{1em}(23)}$$
注：我们一般默认为$\Sigma$为半正定矩阵，这里为了方便计算，我们假设$\Sigma$为正定矩阵。

3.2马氏距离

对于多元高斯分布，我们认为马氏距离L(实数)为：
$$L=(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )\text{\hspace{1em}(24)}$$
由于$\Sigma$为正定矩阵，故可以对其进行特征值分解：
$$\Sigma =U\Lambda U^T;U^{T}U=U{U}^T=I；\Lambda =diag({\lambda }_i);i=1,2,...,N;U=(u_1,u_2,...,u_p)\text{\hspace{1em}(25)}$$
$$\Sigma =U\Lambda U^T=(u_1,u_2,...,u_p)\left(\begin{array}{cccc}{\lambda }_1& & & \\ & {\lambda }_2& & \\ & & ⋮& \\ & & & {\lambda }_p\end{array}\right)\left(\begin{array}{c}{u}_1^T\\ u_2^T\\ ⋮\\ u_p^T\end{array}\right)=\sum _{i=1}^p{u}_i{\lambda }_i{u}_i^T\text{\hspace{1em}(26)}$$
$${\Sigma }^{-1}=(U\Lambda U^T{)}^{-1}=U{\Lambda }^{-1}{U}^T=\sum _{i=1}^p{u}_i\frac{1}{{\lambda }_i}{u}_i^T\text{\hspace{1em}(27)}$$
我们将上式代入到马氏距离L上可得如下：
$$L=(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )=(x-\mu {)}^T\sum _{i=1}^p{u}_i\frac{1}{{\lambda }_i}{u}_i^T(x-\mu )\text{\hspace{1em}(28)}$$
$$L=\frac{1}{{\lambda }_i}\sum _{i=1}^p[(x-\mu {)}^T{u}_i][u_i^T(x-\mu )]\text{\hspace{1em}(29)}$$
我们令$$y_i=\left(\begin{array}{c}{y}_1^T\\ y_2^T\\ ⋮\\ y_N^T\end{array}\right)=(x-\mu {)}^T{u}_i\text{\hspace{1em}(30)}$$
$$L=\sum _{i=1}^p\frac{1}{{\lambda }_i}{y}_i{y}_i^T=\sum _{i=1}^p\frac{y_i^2}{{\lambda }_i}\text{\hspace{1em}(31)}$$
我们假设p=2 ,则$L=\frac{y_1^2}{{\lambda }_1}+\frac{y_2^2}{{\lambda }_2}；表示为一个椭圆$ 如果L取不同的值，我们就可以像等高线一样切除高斯三维的模型，截面形成一个椭圆形状。
$$y_i=(x-\mu {)}^T{u}_i；几何意义解释：$$
$$1.x先向中心偏移后得到的向量(x-\mu {)}^T;2.向量(x-\mu {)}^T在u_i上的投影$$

4.已知联合概率密度p(x,y)，求条件概率密度p(x|y)和边缘概率密度p(x)

4.1样本定义

多变量的高斯分布$X\sim N(\mu ,\Sigma )$,其概率密度函数如下：
$$P(x_i|\theta )=\frac{1}{(2\pi {)}^{\frac{p}{2}}|\Sigma {|}^{\frac{1}{2}}}ex{p}^{(-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu ))}\text{\hspace{1em}(32)}$$
$$X=(x_1,x_2,...,x_N{)}^T={\left(\begin{array}{c}{x}_1^T\\ x_2^T\\ ⋮\\ x_N^T\end{array}\right)}_{P\times 1}={\left(\begin{array}{cccc}{x}_{11}& x_{12}& ...& x_{1p}\\ x_{21}& x_{22}& ...& x_{2p}\\ ⋮& ⋮& ⋮& ⋮\\ x_{N1}& x_{N2}& ...& x_{Np}\end{array}\right)}_{N\times P}\text{\hspace{1em}(33)}$$
$$\Sigma ={\left(\begin{array}{cccc}{\sigma }_{11}& {\sigma }_{12}& ...& {\sigma }_{1p}\\ {\sigma }_{21}& {\sigma }_{22}& ...& {\sigma }_{2p}\\ ⋮& ⋮& ⋮& ⋮\\ {\sigma }_{p1}& {\sigma }_{p2}& ...& {\sigma }_{pp}\end{array}\right)}_{p\times p};\mu ={\left(\begin{array}{c}{\mu }_1^T\\ {\mu }_2^T\\ ⋮\\ {\mu }_p^T\end{array}\right)}_{P\times 1}\text{\hspace{1em}(34)}$$
注：我们一般默认为$\Sigma$为半正定矩阵，这里为了方便计算，我们假设$\Sigma$为正定矩阵。
目的：已知联合概率密度求条件概率密度和边缘概率密度，可描述为已知$x_a为m维度，x_b为n维度；$：
$$已知：X=\left(\begin{array}{c}{x}_a\\ \\ x_b\end{array}\right);m+n=p;\mu =\left(\begin{array}{c}{\mu }_a\\ \\ {\mu }_b\end{array}\right);\Sigma =\left(\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ab}& {\Sigma }_{bb}\end{array}\right);{\Sigma }_{ab}={\Sigma }_{ba}^T\text{\hspace{1em}(35)}$$
$求边缘概率p(x_a),条件概率p(x_b|x_a)$

4.2 引入相关公式-线性组合

$已知:X服从高斯分布X\sim N(\mu ,\Sigma ),Y与X满足线性关系，Y=AX+B$
$结论：Y服从高斯分布Y\sim N(A\mu +B,A\Sigma A^T)$
证明：
$E[Y]=E[AX+B]=AE[X]+B=A\mu +B$
$D[Y]=D[AX+B]=D[AX]+0=AD[X]A^T=A\Sigma A^T$

4.3 独立性证明

$$X若服从高斯分布X\sim N(\mu ,\Sigma ),则MX\perp NX⟺M\Sigma N^T=0\text{\hspace{1em}(36)}$$
证明：
$\because X\sim N(\mu ,\Sigma )$
$\therefore MX\sim N(M\mu ,M\Sigma M^T);NX\sim N(N\mu ,N\Sigma N^T)$

由定义可得：
$COV(MX,NX)$

$=E[(MX-M\mu )(NX-N\mu {)}^T]$

$=E[(M(X-\mu )(X-\mu {)}^T{N}^T]$

$=M\cdot E[((X-\mu )(X-\mu {)}^T]\cdot N^T$

$=M\Sigma N^T$

$\because X\sim N(\mu ,\Sigma ),MX\perp NX$

$\therefore MX和NX相互独立$

$\therefore COV(MX,NX)=0$

$\because COV(MX,NX)=M\Sigma N^T$

$\therefore M\Sigma N^T=0$

我们定义$\Sigma =\left(\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ab}& {\Sigma }_{bb}\end{array}\right),X_{b\cdot a}=X_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{X}_a;我们用矩阵形式表达X_{b\cdot a}，X_a$
$$X_{b\cdot a}=(-{\Sigma }_{ba}{\Sigma }_{aa}^{-1},I)\left(\begin{array}{c}{x}_a\\ \\ x_b\end{array}\right);X_a=(I,0)\left(\begin{array}{c}{x}_a\\ \\ x_b\end{array}\right);\text{\hspace{1em}(37)}$$
$$M=(-{\Sigma }_{ba}{\Sigma }_{aa}^{-1},I);X=\left(\begin{array}{c}{x}_a\\ \\ x_b\end{array}\right);N=(I,0)\text{\hspace{1em}(38)}$$
$\because M\Sigma N^T=(-{\Sigma }_{ba}{\Sigma }_{aa}^{-1},I)\left(\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ab}& {\Sigma }_{bb}\end{array}\right)\left(\begin{array}{c}I\\ \\ 0\end{array}\right)=(0,-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\Sigma }_{ab}+{\Sigma }_{bb})\left(\begin{array}{c}I\\ \\ 0\end{array}\right)=0$
$\therefore MX\perp NX$
$\therefore X_{b\cdot a}\perp X_a$
$$结论：X_{b\cdot a}|X_a=X_{b\cdot a}\text{\hspace{1em}(39)}$$
$我们定义:X_b=X_{b\cdot a}+{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{X}_a$

$$X_b|X_a=X_{b\cdot a}|X_a+{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{X}_a|X_a=X_{b\cdot a}+{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{X}_a=X_b\text{\hspace{1em}(40)}$$
$$结论：X_b|X_a\perp X_b\text{\hspace{1em}(41)}$$
$$\mathbb{E}[X_b|X_a]=\mathbb{E}[X_b]\text{\hspace{1em}(42)}$$
$$\mathbb{D}[X_b|X_a]=\mathbb{D}[X_b]\text{\hspace{1em}(42)}$$

4.4边缘概率$P(x_a)$

由于我们已知:$X=\left(\begin{array}{c}{x}_a\\ \\ x_b\end{array}\right)$;我们构造变量$X_a$满足如下公式：
$$X_a=(I_m,0)\left(\begin{array}{c}{x}_a\\ \\ x_b\end{array}\right)\text{\hspace{1em}(43)}$$
我们可以令
$$A=(I_m,0);X=\left(\begin{array}{c}{x}_a\\ \\ x_b\end{array}\right)\text{\hspace{1em}(44)}$$
$则：X_a=AX$
$已知:X服从高斯分布X\sim N(\mu ,\Sigma ),Y与X满足线性关系，Y=AX+B$
$结论：Y服从高斯分布Y\sim N(A\mu +B,A\Sigma A^T)$
$$\mathbb{E}[X_a]=A\mathbb{E}[X];已知：\mathbb{E}[X]=\left(\begin{array}{c}{\mu }_a\\ {\mu }_b\end{array}\right)\text{\hspace{1em}(45)}$$
$$\mathbb{E}[X_a]=(I_m,0)\left(\begin{array}{c}{\mu }_a\\ {\mu }_b\end{array}\right)={\mu }_a\text{\hspace{1em}(46)}$$
$$\mathbb{D}(X_a)=A\mathbb{D}[X]A^T;已知：\mathbb{D}[X]=\Sigma =\left(\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ab}& {\Sigma }_{bb}\end{array}\right)\text{\hspace{1em}(47)}$$
$$\mathbb{D}(X_a)=A\Sigma A^T=(I_m,0)\left(\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ab}& {\Sigma }_{bb}\end{array}\right)\left(\begin{array}{c}{I}_m\\ 0\end{array}\right)={\Sigma }_{aa}\text{\hspace{1em}(48)}$$
$$结论：X_a\sim N({\mu }_a,{\Sigma }_{aa})$$

4.5条件概率密度$p(x_b|x_a)$

这里我们运用构造法来求，我们需要引入如下构造公式：
$$x_{b\cdot a}=x_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a\text{\hspace{1em}(49)}$$
$${\mu }_{b\cdot a}={\mu }_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\mu }_a\text{\hspace{1em}(50)}$$
$${\Sigma }_{bb\cdot a}={\Sigma }_{bb}-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\Sigma }_{ab}\text{\hspace{1em}(51)}$$
我们可以将$X_{b\cdot a}$用矩阵形式表示：
$$x_{b\cdot a}=(-{\Sigma }_{ba}{\Sigma }_{aa}^{-1},I_n)\left(\begin{array}{c}{x}_a\\ \\ x_b\end{array}\right)\text{\hspace{1em}(52)}$$
$$令:A_1=(-{\Sigma }_{ba}{\Sigma }_{aa}^{-1},I_n);X=\left(\begin{array}{c}{x}_a\\ \\ x_b\end{array}\right);Y=x_{b\cdot a}\text{\hspace{1em}(53)}$$
$$\mathbb{E}[Y]=A_1\mathbb{E}[X]=(-{\Sigma }_{ba}{\Sigma }_{aa}^{-1},I_n)\left(\begin{array}{c}{\mu }_a\\ {\mu }_b\end{array}\right)={\mu }_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\mu }_a={\mu }_{b\cdot a}\text{\hspace{1em}(54)}$$
$$\mathbb{D}[Y]=A_1\mathbb{D}[X]A_1^T=(-{\Sigma }_{ba}{\Sigma }_{aa}^{-1},I_n)\left(\begin{array}{cc}{\Sigma }_{aa}& {\Sigma }_{ab}\\ {\Sigma }_{ab}& {\Sigma }_{bb}\end{array}\right)\left(\begin{array}{c}-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}\\ \\ I_n\end{array}\right)={\Sigma }_{bb}-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\Sigma }_{ab}={\Sigma }_{bb\cdot a}\text{\hspace{1em}(55)}$$
$$Y=x_{b\cdot a}\sim N({\mu }_{b\cdot a},{\Sigma }_{bb\cdot a})\text{\hspace{1em}(56)}$$
由于我们定义了：$x_{b\cdot a}=x_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a;(49公式)；独立性：X_{b\cdot a}|X_a=X_{b\cdot a}$
$$条件性：x_{b\cdot a}|x_a=x_b|x_a-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a|x_a;\text{\hspace{1em}(57)}$$
$$x_{b\cdot a}=x_b|x_a-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a;\text{\hspace{1em}(58)}$$
$$\mathbb{E}[x_{b\cdot a}]=\mathbb{E}[x_b|x_a]-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a\text{\hspace{1em}(59)}$$
$x_a是已知量;可以将{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a看作常量对待$
$$\mathbb{E}[x_b|x_a]=\mathbb{E}[x_{b\cdot a}]+{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a\text{\hspace{1em}(60)}$$
$$\mathbb{E}[x_{b\cdot a}]={\mu }_{b\cdot a}={\mu }_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\mu }_a;\text{\hspace{1em}(61)}$$
$$\mathbb{E}[x_b|x_a]={\mu }_b-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\mu }_a+{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{x}_a\text{\hspace{1em}(62)}$$
$$\mathbb{E}[x_b|x_a]={\mu }_b+{\Sigma }_{ba}{\Sigma }_{aa}^{-1}(x_a-{\mu }_a)\text{\hspace{1em}(63)}$$
$$\mathbb{D}[x_b|x_a]=\mathbb{D}[x_{b\cdot a}]={\Sigma }_{bb}-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\Sigma }_{ab}\text{\hspace{1em}(64)}$$
$$结论：p(x_b|x_a)\sim N[{\mu }_b+{\Sigma }_{ba}{\Sigma }_{aa}^{-1}(x_a-{\mu }_a),{\Sigma }_{bb}-{\Sigma }_{ba}{\Sigma }_{aa}^{-1}{\Sigma }_{ab}]\text{\hspace{1em}(65)}$$
$$结论：p(x_a|x_b)\sim N[{\mu }_a+{\Sigma }_{ab}{\Sigma }_{bb}^{-1}(x_b-{\mu }_b),{\Sigma }_{aa}-{\Sigma }_{ab}{\Sigma }_{bb}^{-1}{\Sigma }_{ba}]\text{\hspace{1em}(65)}$$

5.已知边缘概率密度p(x)和条件概率密度p(y|x)，求边缘概率p(y),条件概率p(x|y)

$已知：p(x)=N(x|\mu ,{\Lambda }^{-1})；p(y|x)=N(Ax+b,L^{-1});求p(y)和p(x|y)？$
$注：协方差矩阵的逆表示精度矩阵{\Lambda }^{-1};变量Y与X服从线性关系Y=Ax+b+ϵ;噪声ϵ独立于X,ϵ\sim N(0,L^{-1})$

5.1求边缘概率p(y)

$$\mathbb{E}[Y]=\mathbb{E}[Ax+b+ϵ]=A\mathbb{E}[X]+b+0=A\mu +b\text{\hspace{1em}(66)}$$
$$\mathbb{D}[Y]=\mathbb{D}[Ax+b+ϵ]=A\mathbb{D}[X]A^T+0+L^{-1}=A{\Lambda }^{-1}{A}^T+L^{-1}\text{\hspace{1em}(66)}$$
$$结论：p(y)\sim N(A\mu +b,A{\Lambda }^{-1}{A}^T+L^{-1})\text{\hspace{1em}(67)}$$

5.2求条件概率p(x|y)

到目前为止，我们已知p(x),p(y|x),p(y);如果我们知道联合概率p(x,y)，那么就可以直接用4中的结论直接求p(x|y)；

5.2.1构造Z

$$令z=\left(\begin{array}{c}x\\ \\ y\end{array}\right);x\sim N(\mu ,{\Lambda }^{-1});y\sim N(A\mu +b,A{\Lambda }^{-1}{A}^T+L^{-1})\text{\hspace{1em}(68)}$$
$$z=\left(\begin{array}{c}x\\ \\ y\end{array}\right)\sim N(\left(\begin{array}{c}\mu \\ \\ A\mu +b\end{array}\right),\left(\begin{array}{cc}{\Lambda }^{-1}& \Delta \\ & \\ {\Delta }^T& A{\Lambda }^{-1}{A}^T+L^{-1}\end{array}\right))\text{\hspace{1em}(69)}$$
$注：\Delta 为关于x,y的协方差，我们可以用协方差定义来求解\Delta :$
$\Delta =Cov(x,y)=E[(x-E[X])(y-E[Y]{)}^T]$
$=E[(x-\mu )([Ax+b+ϵ-A\mu -b{)}^T]$
$=E[(x-\mu )[A(x-\mu )+ϵ{]}^T]$
$=E[(x-\mu )(x-\mu {)}^T{A}^T+(x-\mu ){ϵ}^T]$
$=E[(x-\mu )(x-\mu {)}^T]A^T+E[(x-\mu ){ϵ}^T]]$
$注：由于x独立于ϵ,故E[(x-\mu ){ϵ}^T]=0$
$=E[(x-\mu )(x-\mu {)}^T]A^T$
$=D(x)A^T$
$={\Lambda }^{-1}{A}^T$
$$结论：z=\left(\begin{array}{c}x\\ \\ y\end{array}\right)\sim N(\left(\begin{array}{c}\mu \\ \\ A\mu +b\end{array}\right),\left(\begin{array}{cc}{\Lambda }^{-1}& {\Lambda }^{-1}{A}^T\\ & \\ A{\Lambda }^{-1}& A{\Lambda }^{-1}{A}^T+L^{-1}\end{array}\right))\text{\hspace{1em}(69)}$$

5.2.2求解条件概率p(x|y);

由结论65可得：
$$\mathbb{E}(x|y)=\mu +{\Lambda }^{-1}{A}^T(A{\Lambda }^{-1}{A}^T+L^{-1}{)}^{-1}(y-A\mu -b)\text{\hspace{1em}(70)}$$
$$\mathbb{D}(x|y)={\Lambda }^{-1}-{\Lambda }^{-1}{A}^T(A{\Lambda }^{-1}{A}^T+L^{-1}{)}^{-1}A{\Lambda }^{-1}\text{\hspace{1em}(71)}$$

5.3结论

$$结论：p(x|y)\sim N[\mu +{\Lambda }^{-1}{A}^T(A{\Lambda }^{-1}{A}^T+L^{-1}{)}^{-1}(y-A\mu -b),{\Lambda }^{-1}-{\Lambda }^{-1}{A}^T(A{\Lambda }^{-1}{A}^T+L^{-1}{)}^{-1}A{\Lambda }^{-1}]$$