距离

相对熵、KL散度

熵

​​$H\left(X\right)=-\sum _{i=1}^{n}p\left(x_i\right)\log p\left(x_i\right)$

相随熵

D ( p ∣ ∣ q ) = − ∑ i = 1 n p ( x i ) l o g p ( x i ) q ( x i ) D(p||q)=-\sum\limits_{i=1}^n {p(x_i)log\frac{p(x_i)}{q(x_i)} }