两个多维高斯分布的Kullback-Leibler divergence(KL散度)

最新推荐文章于 2024-02-27 14:36:44 发布

philpy_used

最新推荐文章于 2024-02-27 14:36:44 发布

阅读量3.5k

点赞数 13

分类专栏：统计机器学习文章标签：机器学习深度学习统计学统计模型人工智能

本文链接：https://blog.csdn.net/qq_43613793/article/details/105262433

版权

统计机器学习专栏收录该内容

9 篇文章 3 订阅

订阅专栏

两个高斯分布分别为：
$p(x)=N(x_j;\mu,\sum)\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ \\=\frac{1}{(2\pi)^{\frac{n}{2}}|\sum|^{\frac{1}{2}}}exp\bigg\{{-\frac{1}{2}}(x-\mu)^T(\sum)^{-1} (x-\mu)\bigg\}$
$q(x)=N(x_j;m,L)\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ \\=\frac{1}{(2\pi)^{\frac{n}{2}}|L|^{\frac{1}{2}}}exp\bigg\{{-\frac{1}{2}}(x-m)^TL^{-1} (x-m)\bigg\}$

矩阵迹(tr)的性质：
$tr(\alpha A+\beta B)=\alpha tr(A)+\beta tr(B)......①$ $tr(A)=tr(A^T)......②$ $t r (A B) = t r (B A) . . . . . . ③$ $t r (A B C) = t r (B C A) = t r (C A B) . . . . . . ④ (由 ③ 得)$
一个重要公式： $\lambda^TA\lambda=tr(\lambda^TA\lambda)=tr(A\lambda\lambda^T)......⑤$
多元分布中期望E与协方差 $\sum$ 的性质:
$E(xx^T)=\sum+\mu\mu^T...... ⑥$
证明： $\sum=E\big[(x-\mu)(x-\mu^T)\big] \\=E\big(xx^T-x\mu^T-\mu x^T+\mu\mu^T\big) \\=E\big(xx^T-\mu\mu^T-\mu\mu^T+\mu\mu^T\big) \\=E\big(xx^T\big)-\mu\mu^T \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ \ \ \$
$E\big(x^TAx\big)=tr\big(A\sum\big)+\mu^TA\mu......⑦$
证明：
$E\big(x^TAx\big) \\=E\big[tr(x^TAx)\big] \\=E\big[tr(Axx^T)\big] \\=tr\big[E(Axx^T)\big] \\=tr\big[AE(xx^T)\big] \\=tr\big[A(\sum+\mu\mu^T)\big] \\=tr(A\sum)+tr(A\mu\mu^T) \\=tr(A\sum)+tr(\mu^TA\mu) \\=tr(A\sum)+\mu^TA\mu$

$K L 散度的定义：$
$KL(p||q)=E_p\bigg[log\frac{p(x)}{q(x)}\bigg]$

$\frac{p(x)}{q(x)}=\frac{\frac{1}{(2\pi)^{\frac{n}{2}}|\sum|^{\frac{1}{2}}}exp\bigg\{{-\frac{1}{2}}(x-\mu)^T(\sum)^{-1} (x-\mu)\bigg\}}{\frac{1}{(2\pi)^{\frac{n}{2}}|L|^{\frac{1}{2}}}exp\bigg\{{-\frac{1}{2}}(x-m)^TL^{-1} (x-m)\bigg\}} \\=(\frac{|L|}{|\sum|})^{\frac{1}{2}}exp\bigg\{{-\frac{1}{2}}(x-\mu)^T(\sum)^{-1} (x-\mu)-\big[{-\frac{1}{2}}(x-m)^TL^{-1} (x-m)\big]\bigg\} \\=(\frac{|L|}{|\sum|})^{\frac{1}{2}}exp\bigg\{\frac{1}{2}\big[(x-m)^TL^{-1} (x-m)-(x-\mu)^T(\sum)^{-1} (x-\mu)\big]\bigg\}$

$log\frac{p(x)}{q(x)}=log\Bigg((\frac{|L|}{|\sum|})^{\frac{1}{2}}exp\bigg\{\frac{1}{2}\big[(x-m)^TL^{-1} (x-m)-(x-\mu)^T(\sum)^{-1} (x-\mu)\big]\bigg\}\Bigg) \\=\frac{1}{2}log\frac{|L|}{|\sum|}+\frac{1}{2}\big[(x-m)^TL^{-1} (x-m)-(x-\mu)^T(\sum)^{-1} (x-\mu)\big]$

$E_p\bigg[log\frac{p(x)}{q(x)}\bigg] \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ \\=E_p\bigg(\frac{1}{2}log\frac{|L|}{|\sum|}+\frac{1}{2}\big[(x-m)^TL^{-1} (x-m)-(x-\mu)^T(\sum)^{-1} (x-\mu)\big]\bigg) \\=\frac{1}{2}E_p\bigg(log\frac{|L|}{|\sum|}\bigg)+\frac{1}{2}E_p\bigg((x-m)^TL^{-1} (x-m)-(x-\mu)^T(\sum)^{-1} (x-\mu)\bigg) \\=\frac{1}{2}log\frac{|L|}{|\sum|}+\frac{1}{2}E_p\bigg(tr\big[L^{-1} (x-m)(x-m)^T\big]-tr\big[(\sum)^{-1} (x-\mu)(x-\mu)^T\big]\bigg)......(性质⑤) \\=\frac{1}{2}log\frac{|L|}{|\sum|}+\frac{1}{2}tr\bigg(E_p\big[L^{-1} (x-m)(x-m)^T\big]\bigg)\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ \\-\frac{1}{2}tr\bigg(E_p\big[(\sum)^{-1} (x-\mu)(x-\mu)^T\big]\bigg) ......(性质①) \\=\frac{1}{2}log\frac{|L|}{|\sum|}+\frac{1}{2}tr\bigg(E_p\big[ L^{-1}(xx^T-mx^T-xm^T+mm^T)\big]\bigg)\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ \\-\frac{1}{2}tr\bigg((\sum)^{-1}E_p\big[(\sum)^{-1} (x-\mu)(x-\mu)^T\big]\bigg) \\=\frac{1}{2}log\frac{|L|}{|\sum|}+\frac{1}{2}tr\bigg(L^{-1}\big[ E_p(xx^T-mx^T-xm^T+mm^T)\big]\bigg)\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ \\-\frac{1}{2}tr\big((\sum)^{-1}\sum\big) \\=\frac{1}{2}log\frac{|L|}{|\sum|}+\frac{1}{2}tr\bigg(L^{-1}\big[\underbrace{\sum+\mu\mu^T}_{性质⑥}-mx^T-xm^T+mm^T\big]\bigg)-\frac{n}{2} \\=\frac{1}{2}\Bigg\{log\frac{|L|}{|\sum|}-n+tr\big(L^{-1}\sum\big)+tr\big(L^{-1}[\mu\mu^T-mx^T-xm^T+mm^T]\big)\Bigg\} \\=\frac{1}{2}\Bigg\{log\frac{|L|}{|\sum|}-n+tr\big(L^{-1}\sum\big)+tr\big(L^{-1}\mu\mu^T-L^{-1}mx^T-L^{-1}xm^T+L^{-1}mm^T\big)\Bigg\} \\=\frac{1}{2}\Bigg\{log\frac{|L|}{|\sum|}-n+tr\big(L^{-1}\sum\big)+tr\big(\mu^TL^{-1}\mu-2x^TL^{-1}m+m^TL^{-1}m\big)\Bigg\} \\=\frac{1}{2}\Bigg\{log\frac{|L|}{|\sum|}-n+tr\big(L^{-1}\sum\big)+tr\big(L^{-1}\mu\mu^T-L^{-1}mx^T-L^{-1}xm^T+L^{-1}mm^T\big)\Bigg\} \\=\frac{1}{2}\Bigg\{log\frac{|L|}{|\sum|}-n+tr\big(L^{-1}\sum\big)+\big(x-m\big)^TL^{-1}\big(x-m\big)\Bigg\}$