多变量高斯分布之间的KL散度（KL Divergence）

最新推荐文章于 2024-08-08 17:18:15 发布

wangpeng138375

最新推荐文章于 2024-08-08 17:18:15 发布

阅读量1.7w

点赞数 36

分类专栏：理论机器学习文章标签：变分-高斯分布-KL

本文链接：https://blog.csdn.net/wangpeng138375/article/details/78060753

版权

理论同时被 2 个专栏收录

7 篇文章 0 订阅

订阅专栏

机器学习

6 篇文章 0 订阅

订阅专栏

单变量高斯分布的概率密度函数如下（均值：u，方差： $\sigma$ ）：

N (x | u, σ) = 1 ( 2 π σ 2 ) 1 / 2 exp {- 1 2 σ 2 (x - u) 2}

$N(x|u,{\sigma }) = {1 \over {{{(2\pi {\sigma ^2})}^{1/2}}}}\exp \{ - {1 \over {2{\sigma ^2}}}{(x - u)^2}\}$
多变量高斯分布（假设n维）的概率密度函数如下（均值：u，协方差矩阵：

Σ $\Sigma$ ）：

N (x | u, Σ) = 1 ( 2 π ) n / 2 | Σ | 1 / 2 exp {- 1 2 (x - u) T Σ - 1 (x - u)}

$N(x|u,\Sigma ) = {1 \over {{{(2\pi )}^{n/2}}{{\left| \Sigma \right|}^{1/2}}}}\exp \{ - {1 \over 2}{(x - u)^T}{\Sigma ^{ - 1}}(x - u)\}$

在公式推导之前，首先介绍一些用到的性质。

矩阵的迹的性质：
（1） $tr(\alpha A + \beta B) = \alpha tr(A) + \beta tr(B)$
（2） $tr(A) = tr({A^T})$
（3） $tr(AB) = tr(BA)$
根据性质（3）可以得到性质（4）
（4） $tr(ABC) = tr(CAB) = tr(BCA)$
在推导公式过程中，使用到的一个重要的trick如下：
对于列向量 $\lambda$ ，公式 ${\lambda ^T}A\lambda$ 的结果是一个标量，所以：

$λ T A λ = t r (λ T A λ) = t r (A λ λ T)$ ${\lambda ^T}A\lambda = tr({\lambda ^T}A\lambda ) = tr(A\lambda {\lambda ^T})$
多变量分布中期望 $E$ 与协方差 $\Sigma$ 的性质：
（1） $E\left[ {x{x^T}} \right] = \Sigma + u{u^T}$
证明：
$Σ = E [(x - u) (x - u) T] = E [x x T - x u T - u x T + u u T] = E [x x T] - u u T - u u T + u u T = E [x x T] - u u T$ $\eqalign{ & \Sigma \cr & = E\left[ {\left( {x - u} \right){{\left( {x - u} \right)}^T}} \right] \cr & = E\left[ {x{x^T} - x{u^T} - u{x^T} + u{u^T}} \right] \cr & = E\left[ {x{x^T}} \right] - u{u^T} - u{u^T} + u{u^T} \cr & = E\left[ {x{x^T}} \right] - u{u^T} \cr}$
（2） $E\left( {{x^T}Ax} \right) = tr(A\Sigma ) + {u^T}Au$
证明：
因为 ${{x^T}Ax}$ 的结果是一个标量，利用前面提到的trick，可得：
$E (x T A x) = E [t r (x T A x)] = E [t r (A x x T)] = t r [E (A x x T)] = t r [A E (x x T)] = t r [A (Σ + u u T)] = t r (A Σ) + t r (A u u T) = t r (A Σ) + t r (u T A u) = t r (A Σ) + u T A u$ $\eqalign{ & E\left( {{x^T}Ax} \right) \cr & = E\left[ {tr({x^T}Ax)} \right] \cr & = E\left[ {tr(Ax{x^T})} \right] \cr & = tr\left[ {E\left( {Ax{x^T}} \right)} \right] \cr & = tr\left[ {AE\left( {x{x^T}} \right)} \right] \cr & = tr\left[ {A(\Sigma + u{u^T})} \right] \cr & = tr(A\Sigma ) + tr(Au{u^T}) \cr & = tr(A\Sigma ) + tr({u^T}Au) \cr & = tr(A\Sigma ) + {u^T}Au \cr}$

这里主要讲述多变量高斯分布的KL散度。
连个分布和的的KL散度定义如下:

D K L (P 1 | | P 2) = E P 1 [log P 1 P 2]

${D_{KL}}({P_1}||{P_2}) = {E_{{P_1}}}\left[ {\log {{{P_1}} \over {{P_2}}}} \right]$

D K L (P 1 | | P 2) = E P 1 [log P 1 - log P 2] = 1 2 E P 1 [- log | Σ 1 | - (x - u 1) T Σ - 1 1 (x - u 1) + log | Σ 2 | + (x - u 2) T Σ - 1 2 (x - u 2)] = 1 2 log | Σ 2 | | Σ 1 | + 1 2 E P 1 [- (x - u 1) T Σ - 1 1 (x - u 1) + (x - u 2) T Σ - 1 2 (x - u 2)] = 1 2 log | Σ 2 | | Σ 1 | + 1 2 E P 1 {- t r [Σ - 1 1 (x - u 1) (x - u 1) T] + t r [Σ - 1 2 (x - u 2) (x - u 2) T]} = 1 2 log | Σ 2 | | Σ 1 | + 1 2 E P 1 {- t r [Σ - 1 1 (x - u 1) (x - u 1) T]} + 1 2 E P 1 {t r [Σ - 1 2 (x - u 2) (x - u 2) T]} = 1 2 log | Σ 2 | | Σ 1 | - 1 2 t r {E P 1 [Σ - 1 1 (x - u 1) (x - u 1) T]} + 1 2 t r {E P 1 [Σ - 1 2 (x - u 2) (x - u 2) T]} = 1 2 log | Σ 2 | | Σ 1 | - 1 2 t r {Σ - 1 1 E P 1 [(x - u 1) (x - u 1) T]} + 1 2 t r {E P 1 [Σ - 1 2 (x x T - u 2 x T - x u T 2 + u 2 u T 2)]} = 1 2 log | Σ 2 | | Σ 1 | - 1 2 t r {Σ - 1 1 Σ 1} + 1 2 t r {Σ - 1 2 E P 1 (x x T - u 2 x T - x u T 2 + u 2 u T 2)} = 1 2 log | Σ 2 | | Σ 1 | - 1 2 n + 1 2 t r {Σ - 1 2 (Σ 1 + u 1 u T 1 - u 2 u T 1 - u 1 u T 2 + u 2 u T 2)} - - - 这 里 利 用 了 E [x x T] = Σ + u u T = 1 2 {log | Σ 2 | | Σ 1 | - n + t r (Σ - 1 2 Σ 1) + t r {Σ - 1 2 (u 1 u T 1 - u 2 u T 1 - u 1 u T 2 + u 2 u T 2)}} = 1 2 {log | Σ 2 | | Σ 1 | - n + t r (Σ - 1 2 Σ 1) + t r {Σ - 1 2 u 1 u T 1 - Σ - 1 2 u 2 u T 1 - Σ - 1 2 u 1 u T 2 + Σ - 1 2 u 2 u T 2}} = 1 2 {log | Σ 2 | | Σ 1 | - n + t r (Σ - 1 2 Σ 1) + t r {u T 1 Σ - 1 2 u 1 - 2 u T 1 Σ - 1 2 u 2 + u T 2 Σ - 1 2 u 2}} = 1 2 {log | Σ 2 | | Σ 1 | - n + t r (Σ - 1 2 Σ 1) + (u 2 - u 1) T Σ - 1 2 (u 2 - u 1)}

$\eqalign{ & {D_{KL}}({P_1}||{P_2}) \cr & = {E_{{P_1}}}\left[ {\log {P_1} - \log {P_2}} \right] \cr & = {1 \over 2}{E_{{P_1}}}\left[ { - \log \left| {{\Sigma _1}} \right| - {{(x - {u_1})}^T}\Sigma _1^{ - 1}(x - {u_1}) + \log \left| {{\Sigma _2}} \right| + {{(x - {u_2})}^T}\Sigma _2^{ - 1}(x - {u_2})} \right] \cr & = {1 \over 2}\log {{\left| {{\Sigma _2}} \right|} \over {\left| {{\Sigma _1}} \right|}} + {1 \over 2}{E_{{P_1}}}\left[ { - {{(x - {u_1})}^T}\Sigma _1^{ - 1}(x - {u_1}) + {{(x - {u_2})}^T}\Sigma _2^{ - 1}(x - {u_2})} \right] \cr & = {1 \over 2}\log {{\left| {{\Sigma _2}} \right|} \over {\left| {{\Sigma _1}} \right|}} + {1 \over 2}{E_{{P_1}}}\left\{ { - tr\left[ {\Sigma _1^{ - 1}(x - {u_1}){{(x - {u_1})}^T}} \right] + tr\left[ {\Sigma _2^{ - 1}(x - {u_2}){{(x - {u_2})}^T}} \right]} \right\} \cr & = {1 \over 2}\log {{\left| {{\Sigma _2}} \right|} \over {\left| {{\Sigma _1}} \right|}} + {1 \over 2}{E_{{P_1}}}\left\{ { - tr\left[ {\Sigma _1^{ - 1}(x - {u_1}){{(x - {u_1})}^T}} \right]} \right\} + {1 \over 2}{E_{{P_1}}}\left\{ {tr\left[ {\Sigma _2^{ - 1}(x - {u_2}){{(x - {u_2})}^T}} \right]} \right\} \cr & = {1 \over 2}\log {{\left| {{\Sigma _2}} \right|} \over {\left| {{\Sigma _1}} \right|}} - {1 \over 2}tr\left\{ {{E_{{P_1}}}\left[ {\Sigma _1^{ - 1}(x - {u_1}){{(x - {u_1})}^T}} \right]} \right\} + {1 \over 2}tr\left\{ {{E_{{P_1}}}\left[ {\Sigma _2^{ - 1}(x - {u_2}){{(x - {u_2})}^T}} \right]} \right\} \cr & = {1 \over 2}\log {{\left| {{\Sigma _2}} \right|} \over {\left| {{\Sigma _1}} \right|}} - {1 \over 2}tr\left\{ {\Sigma _1^{ - 1}{E_{{P_1}}}\left[ {(x - {u_1}){{(x - {u_1})}^T}} \right]} \right\} + {1 \over 2}tr\left\{ {{E_{{P_1}}}\left[ {\Sigma _2^{ - 1}(x{x^T} - {u_2}{x^T} - xu_2^T + {u_2}u_2^T)} \right]} \right\} \cr & = {1 \over 2}\log {{\left| {{\Sigma _2}} \right|} \over {\left| {{\Sigma _1}} \right|}} - {1 \over 2}tr\left\{ {\Sigma _1^{ - 1}{\Sigma _1}} \right\} + {1 \over 2}tr\left\{ {\Sigma _2^{ - 1}{E_{{P_1}}}(x{x^T} - {u_2}{x^T} - xu_2^T + {u_2}u_2^T)} \right\} \cr & = {1 \over 2}\log {{\left| {{\Sigma _2}} \right|} \over {\left| {{\Sigma _1}} \right|}} - {1 \over 2}n + {1 \over 2}tr\left\{ {\Sigma _2^{ - 1}({\Sigma _1} + {u_1}u_1^T - {u_2}u_1^T - {u_1}u_2^T + {u_2}u_2^T)} \right\} - - -这里利用了 E\left[ {x{x^T}} \right] = \Sigma + u{u^T} \cr & = {1 \over 2}\left\{ {\log {{\left| {{\Sigma _2}} \right|} \over {\left| {{\Sigma _1}} \right|}} - n + tr(\Sigma _2^{ - 1}{\Sigma _1}) + tr\left\{ {\Sigma _2^{ - 1}({u_1}u_1^T - {u_2}u_1^T - {u_1}u_2^T + {u_2}u_2^T)} \right\}} \right\} \cr & = {1 \over 2}\left\{ {\log {{\left| {{\Sigma _2}} \right|} \over {\left| {{\Sigma _1}} \right|}} - n + tr(\Sigma _2^{ - 1}{\Sigma _1}) + tr\left\{ {\Sigma _2^{ - 1}{u_1}u_1^T - \Sigma _2^{ - 1}{u_2}u_1^T - \Sigma _2^{ - 1}{u_1}u_2^T + \Sigma _2^{ - 1}{u_2}u_2^T} \right\}} \right\} \cr & = {1 \over 2}\left\{ {\log {{\left| {{\Sigma _2}} \right|} \over {\left| {{\Sigma _1}} \right|}} - n + tr(\Sigma _2^{ - 1}{\Sigma _1}) + tr\left\{ {u_1^T\Sigma _2^{ - 1}{u_1} - 2u_1^T\Sigma _2^{ - 1}{u_2} + u_2^T\Sigma _2^{ - 1}{u_2}} \right\}} \right\} \cr & = {1 \over 2}\left\{ {\log {{\left| {{\Sigma _2}} \right|} \over {\left| {{\Sigma _1}} \right|}} - n + tr(\Sigma _2^{ - 1}{\Sigma _1}) + {{({u_2} - {u_1})}^T}\Sigma _2^{ - 1}({u_2} - {u_1})} \right\} \cr}$