高斯分布的KL散度

两个高斯分布的KL散度其实很简单，只要找到合适的方法。

一. 一维高斯分布
KL散度的定义为：
$$\begin{array}{rl}KL(\mathcal{N}({\mu }_1,{\sigma }_1^2)||\mathcal{N}({\mu }_2,{\sigma }_2^2))& ={\int }_x\frac{1}{\sqrt{2\pi }{\sigma }_1}{e}^{-\frac{(x-{\mu }_1{)}^2}{2{\sigma }_1^2}}\log \frac{\frac{1}{\sqrt{2\pi }{\sigma }_1}{e}^{-\frac{(x-{\mu }_1{)}^2}{2{\sigma }_1^2}}}{\frac{1}{\sqrt{2\pi }{\sigma }_2}{e}^{-\frac{(x-{\mu }_2{)}^2}{2{\sigma }_2^2}}}dx\\ & ={\int }_x\frac{1}{\sqrt{2\pi }{\sigma }_1}{e}^{-\frac{(x-{\mu }_1{)}^2}{2{\sigma }_1^2}}[\log \frac{{\sigma }_2}{{\sigma }_1}-\frac{(x-{\mu }_1{)}^2}{2{\sigma }_1^2}+\frac{(x-{\mu }_2{)}^2}{2{\sigma }_2^2}]dx\end{array}$$

第一项很简单，用全积分为1的性质即可：
$$\begin{array}{r}\log \frac{{\sigma }_2}{{\sigma }_1}{\int }_x\frac{1}{\sqrt{2\pi }{\sigma }_1}{e}^{-\frac{(x-{\mu }_1{)}^2}{2{\sigma }_1^2}}dx=\log \frac{{\sigma }_2}{{\sigma }_1}\end{array}$$

第二项需要分辨出积分项为方差：
$$\begin{array}{r}-\frac{1}{2{\sigma }_1^2}{\int }_x(x-{\mu }_1{)}^2\frac{1}{\sqrt{2\pi }{\sigma }_1}{e}^{-\frac{(x-{\mu }_1{)}^2}{2{\sigma }_1^2}}dx=-\frac{1}{2{\sigma }_1^2}{\sigma }_1^2=-\frac{1}{2}\end{array}$$

第三项的积分内部分别是均方值、均值和常数，因此可以得到：
$$\begin{array}{rl}\frac{1}{2{\sigma }_2^2}{\int }_x(x-{\mu }_2{)}^2\frac{1}{\sqrt{2\pi }{\sigma }_1}{e}^{-\frac{(x-{\mu }_1{)}^2}{2{\sigma }_1^2}}dx& =\frac{1}{2{\sigma }_2^2}{\int }_x(x^2-2{\mu }_{2}x+{\mu }_2^2)\frac{1}{\sqrt{2\pi }{\sigma }_1}{e}^{-\frac{(x-{\mu }_1{)}^2}{2{\sigma }_1^2}}dx\\ & =\frac{{\sigma }_1^2+{\mu }_1^2-2{\mu }_1{\mu }_2+{\mu }_2^2}{2{\sigma }_2^2}=\frac{{\sigma }_1^2+({\mu }_1-{\mu }_2{)}^2}{2{\sigma }_2^2}\end{array}$$
也可以用一个小技巧来化简，其中第一项为方差，第二项为奇函数全积分为0，第三项为常数可以提取为系数：
$$\begin{array}{rl}\frac{1}{2{\sigma }_2^2}{\int }_x(x-{\mu }_2{)}^2\frac{1}{\sqrt{2\pi }{\sigma }_1}{e}^{-\frac{(x-{\mu }_1{)}^2}{2{\sigma }_1^2}}dx& =\frac{1}{2{\sigma }_2^2}{\int }_x[(x-{\mu }_1{)}^2+2({\mu }_1-{\mu }_2)(x-{\mu }_1)+({\mu }_1-{\mu }_2{)}^2]\frac{1}{\sqrt{2\pi }{\sigma }_1}{e}^{-\frac{(x-{\mu }_1{)}^2}{2{\sigma }_1^2}}dx\\ & =\frac{{\sigma }_1^2+({\mu }_1-{\mu }_2{)}^2}{2{\sigma }_2^2}\end{array}$$

整理最终结果，两个高斯分布的KL散度为：
$$KL(\mathcal{N}({\mu }_1,{\sigma }_1^2)||\mathcal{N}({\mu }_2,{\sigma }_2^2))=\log \frac{{\sigma }_2}{{\sigma }_1}-\frac{1}{2}+\frac{{\sigma }_1^2+({\mu }_1-{\mu }_2{)}^2}{2{\sigma }_2^2}$$

二. 多元高斯分布

$$\begin{array}{r}\mathcal{N}(x|\mu ,\Sigma )=\frac{1}{(2\pi {)}^{\frac{K}{2}}|\Sigma {|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(x-\mu {)}^T{\Sigma }^{-1}(x-\mu )}\end{array}$$

$$\begin{array}{rl}KL(\mathcal{N}(x|{\mu }_1,{\Sigma }_1)||\mathcal{N}(x|{\mu }_2,{\Sigma }_2))& ={\int }_{x_1}\cdots {\int }_{x_K}\frac{1}{(2\pi {)}^{\frac{K}{2}}|{\Sigma }_1{|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)}\log \frac{\frac{1}{(2\pi {)}^{\frac{K}{2}}|{\Sigma }_1{|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)}}{\frac{1}{(2\pi {)}^{\frac{K}{2}}|{\Sigma }_2{|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(x-{\mu }_2{)}^T{\Sigma }_2^{-1}(x-{\mu }_2)}}d{x}_1\cdots d{x}_K\\ & ={\int }_{x_1}\cdots {\int }_{x_K}\frac{1}{(2\pi {)}^{\frac{K}{2}}|{\Sigma }_1{|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(x-{\mu }_1{)}^T{\Sigma }^{-1}(x-{\mu }_1)}[\frac{1}{2}\log \frac{|{\Sigma }_2|}{|{\Sigma }_1|}-\frac{1}{2}(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)+\frac{1}{2}(x-{\mu }_2{)}^T{\Sigma }_2^{-1}(x-{\mu }_2)]d{x}_1\cdots d{x}_K\end{array}$$
同样分别计算三项的结果，第一项：
$$\begin{array}{r}\frac{1}{2}\log \frac{|{\Sigma }_2|}{|{\Sigma }_1|}{\int }_{x_1}\cdots {\int }_{x_K}\frac{1}{(2\pi {)}^{\frac{K}{2}}|{\Sigma }_1{|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)}d{x}_1\cdots d{x}_K=\frac{1}{2}\log \frac{|{\Sigma }_2|}{|{\Sigma }_1|}\end{array}$$
第二项：
$$\begin{array}{rl}& -\frac{1}{2}{\int }_{x_1}\cdots {\int }_{x_K}\frac{1}{(2\pi {)}^{\frac{K}{2}}|{\Sigma }_1{|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)}(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)d{x}_1\cdots d{x}_K\end{array}$$
${\Sigma }_1$为半正定对称矩阵，设${\Sigma }_1^{-1}=U^{T}U$, $y=U(x-{\mu }_1)$，由于线性变换矩阵就是雅克比矩阵，因此
$$d{y}_1\cdots d{y}_K=|U|d{x}_1\cdots d{x}_K$$
由$|{\Sigma }_1^{-1}|=|U{|}^2$，可知$|{\Sigma }_1^{-\frac{1}{2}}|=|{\Sigma }_1{|}^{-\frac{1}{2}}=|U|$，因此
$$\begin{array}{rl}& -\frac{1}{2|{\Sigma }_1{|}^{\frac{1}{2}}}{\int }_{y_1}\cdots {\int }_{y_K}\frac{1}{(2\pi {)}^{\frac{K}{2}}}{e}^{-\frac{1}{2}{y}^{T}y}{y}^{T}y|U{|}^{-1}d{y}_1\cdots d{y}_K\\ & =-\frac{1}{2|{\Sigma }_1{|}^{\frac{1}{2}}}|{\Sigma }_1{|}^{\frac{1}{2}}\cdot K=-\frac{K}{2}\end{array}$$

第三项需要利用一个小技巧：
$$x^{T}Ax=tr(Ax{x}^T)$$

$$\begin{array}{rl}& \frac{1}{2}{\int }_{x_1}\cdots {\int }_{x_K}\frac{1}{(2\pi {)}^{\frac{K}{2}}|{\Sigma }_1{|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)}(x-{\mu }_2{)}^T{\Sigma }_2^{-1}(x-{\mu }_2)d{x}_1\cdots d{x}_K\\ & =\frac{1}{2}{\int }_{x_1}\cdots {\int }_{x_K}\frac{1}{(2\pi {)}^{\frac{K}{2}}|{\Sigma }_1{|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)}tr[{\Sigma }_2^{-1}(x-{\mu }_2)(x-{\mu }_2{)}^T]d{x}_1\cdots d{x}_K\\ & =\frac{1}{2}tr[{\Sigma }_2^{-1}{\int }_{x_1}\cdots {\int }_{x_K}\frac{1}{(2\pi {)}^{\frac{K}{2}}|{\Sigma }_1{|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)}(x-{\mu }_2)(x-{\mu }_2{)}^T]d{x}_1\cdots d{x}_K\\ & =\frac{1}{2}tr[{\Sigma }_2^{-1}{\int }_{x_1}\cdots {\int }_{x_K}\frac{1}{(2\pi {)}^{\frac{K}{2}}|{\Sigma }_1{|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)}(x{x}^T-{\mu }_2{x}^T-x{\mu }_2^T+{\mu }_2{\mu }_2^T)]d{x}_1\cdots d{x}_K\end{array}$$
其中积分之后第一项为均方值，第二、三项为均值，第三项为常数：
$$\begin{array}{rl}& \frac{1}{2}tr[{\Sigma }_2^{-1}{\int }_{x_1}\cdots {\int }_{x_K}\frac{1}{(2\pi {)}^{\frac{K}{2}}|{\Sigma }_1{|}^{\frac{1}{2}}}{e}^{-\frac{1}{2}(x-{\mu }_1{)}^T{\Sigma }_1^{-1}(x-{\mu }_1)}(x{x}^T-{\mu }_2{x}^T-x{\mu }_2^T+{\mu }_2{\mu }_2^T)]d{x}_1\cdots d{x}_K\\ & =\frac{1}{2}tr[{\Sigma }_2^{-1}({\Sigma }_1+{\mu }_1{\mu }_1^T-{\mu }_2{\mu }_1^T-{\mu }_1{\mu }_2^T+{\mu }_2{\mu }_2^T)]\\ & =\frac{1}{2}[tr({\Sigma }_2^{-1}{\Sigma }_1)+tr({\Sigma }_2^{-1}({\mu }_1-{\mu }_2)({\mu }_1-{\mu }_2{)}^T)]\\ & =\frac{1}{2}[tr({\Sigma }_2^{-1}{\Sigma }_1)+({\mu }_1-{\mu }_2{)}^T{\Sigma }_2^{-1}({\mu }_1-{\mu }_2)]\end{array}$$

整理最终结果，两个高斯分布的KL散度为：
$$\begin{array}{r}KL(\mathcal{N}(x|{\mu }_1,{\Sigma }_1)||\mathcal{N}(x|{\mu }_2,{\Sigma }_2))=\frac{1}{2}[\log \frac{|{\Sigma }_2|}{|{\Sigma }_1|}-K+tr({\Sigma }_2^{-1}{\Sigma }_1)+({\mu }_1-{\mu }_2{)}^T{\Sigma }_2^{-1}({\mu }_1-{\mu }_2)]\end{array}$$