KL散度，互信息和JSD

最新推荐文章于 2022-09-22 19:51:32 发布

Pre.Real

最新推荐文章于 2022-09-22 19:51:32 发布

阅读量792

点赞数

文章标签：信息熵

本文链接：https://blog.csdn.net/weixin_43691167/article/details/122439973

版权

KL 散度公式
$\begin{aligned}D(p\|q) &=\sum_{x \in X} p(x) \log \frac{p(x)}{q(x)} \\&\left.=E\left[\log \frac{p(X)}{q(X)}\right)\right]\end{aligned}$

互信息公式
$\begin{aligned} I(X, Y) &=H(X)-H(X \mid Y) \\ &=\sum_{x, y} p(x, y) \log \frac{p(y \mid x)}{p(y))} \end{aligned}$

KL与互信息
$\begin{aligned} I(X, Y) &=H(X)-H(X \mid Y) \\ &=H(Y)-H(Y \mid X) \\ &=D(p(x, y) \| p(x) p(y)) \\ &=E\left[\log \frac{p(x, y)}{p(x) p(y)}\right] \end{aligned}$

Jensen-Shannon Divergence公式
$\begin{aligned} &\quad \operatorname{JSD}(P \| Q)=\frac{1}{2} D(P \| M)+\frac{1}{2} D(Q \| M) \\ &\text { where } M=\frac{1}{2}(P+Q) \end{aligned}$

tensorflow KL损失函数

import numpy as np
import pandas as pd
import tensorflow as tf


y_true = np.random.randint(0, 2, size=(2, 3)).astype(np.float64)
y_pred = np.random.random(size=(2, 3))
loss = tf.keras.losses.kullback_leibler_divergence(y_true, y_pred)
assert loss.shape == (2,)
y_true = tf.keras.backend.clip(y_true, 1e-7, 1)
y_pred = tf.keras.backend.clip(y_pred, 1e-7, 1)
assert np.array_equal(
    loss.numpy(), np.sum(y_true * np.log(y_true / y_pred), axis=-1))

print(y_true * np.log(y_true / y_pred))