Kullback-Leibler (KL) loss

Kullback-Leibler($\mathrm{K}\mathrm{L}$) loss
（离散）For discrete probability distributions $F(x)$ and $G(x)$, the Kullback-Leibler ($\mathrm{K}\mathrm{L}$) loss from $F(x)$ to $G(x)$ is defined[5] to be
K L { F ( x ) ∥ G ( x ) } = ∑ i = 1 n F ( x ) log ⁡ F ( x ) G ( x ) . \mathrm {KL}\{F(x)\|G(x)\} = \sum_{i=1}^nF(x)\log\frac{F(x)}{G(x)}. KL{F(x)∥G(x)}=i=1∑n​F(x)logG(x)F(x)​.
（连续）For distributions $F(x)$ and $G(x)$ of a continuous random variable, the Kullback–Leibler($\mathrm{K}\mathrm{L}$) loss is defined to be
K L { F ( x ) ∥ G ( x ) } = ∫ − ∞ ∞ f ( x ) log ⁡ f ( x ) g ( x ) d x \mathrm {KL}\{F(x)\|G(x)\} = \int_{-\infty}^{\infty}f(x)\log\frac{f(x)}{g(x)}dx KL{F(x)∥G(x)}=∫−∞∞​f(x)logg(x)f(x)​dx
where $f(x)$ and $g(x)$ is the densities function of $F(x)$ and $G(x)$.

The Kullback–Leibler loss is always non-negative(始终非负), that is
K L { F ( x ) ∥ G ( x ) } ⩾ 0. \mathrm {KL}\{F(x)\|G(x)\}\geqslant0. KL{F(x)∥G(x)}⩾0.
The Kullback–Leibler($\mathrm{K}\mathrm{L}$) loss K L { F ( x ) ∥ G ( x ) } \mathrm {KL}\{F(x)\|G(x)\} KL{F(x)∥G(x)} is convex(凸的) in the pair of probability mass functions $(f,g)$, i.e. if $(f_1,g_1)$ and $(f_2,g_2)$ are two pairs of probability mass functions, then K L { λ f 1 + ( 1 − λ ) f 2 ∥ λ g 1 + ( 1 − λ ) g 2 } ≤ λ K L ( f 1 ∥ g 1 ) + ( 1 − λ ) K L ( f 2 ∥ g 2 ) {\mathrm {KL}\{\lambda f_{1}+(1-\lambda )f_{2}\|\lambda g_{1}+(1-\lambda )g_{2}\}\leq \lambda \mathrm {KL} (f_{1}\|g_{1})+(1-\lambda )\mathrm {KL} (f_{2}\|g_{2})} KL{λf1​+(1−λ)f2​∥λg1​+(1−λ)g2​}≤λKL(f1​∥g1​)+(1−λ)KL(f2​∥g2​) for $0\le \lambda \le 1$.

eg: Multivariate normal distributions
Suppose that we have two multivariate normal distributions, with means ${\mu }_0,{\mu }_1$ and with (nonsingular) covariance matrices ${\Sigma }_0,{\Sigma }_1$. If the two distributions have the same dimension, k, then the Kullback–Leibler($\mathrm{K}\mathrm{L}$ ) loss between the distributions is as follows:
$$\mathrm{K}\mathrm{L}({\mathcal{N}}_0\Vert {\mathcal{N}}_1)=\frac{1}{2}\left\{\mathrm{t}\mathrm{r}\left({\Sigma }_1^{-1}{\Sigma }_0\right)+{\left({\mu }_1-{\mu }_0\right)}^{\text{T}}{\Sigma }_1^{-1}({\mu }_1-{\mu }_0)-k+\log \left(\frac{\det {\Sigma }_1}{\det {\Sigma }_0}\right)\right\}.$$