李宏毅深度学习-Improved GAN

1. What is f-divergence ?

《f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization》, NIPS, 2016

You can use any f-divergence!
$P$ and $Q$ are two distributions. $p(x)$ and $q(x)$ are the probability of sampling $x$.
$$D_f(P||Q)=\int_x q(x)f(\frac{p(x)}{q(x)})dx$$
$f$ is convex, $f(1)=0$

①if $f(x)=xlogx$ ——(KL divergence)

$$D_f(P||Q)=\int_x q(x)\frac{p(x)}{q(x)}log(\frac{p(x)}{q(x)})dx=\int_x p(x)log(\frac{p(x)}{q(x)})dx$$
②if $f(x)=-logx$ ——( Reverse KL divergence)
$$D_f(P||Q)=\int_x q(x)(-log(\frac{p(x)}{q(x)}))dx=\int_x q(x)log(\frac{q(x)}{p(x)})dx$$
②if $f(x)=(x-1)^2$ ——( Chi Square divergence)
$$D_f(P||Q)=\int_x q(x)(\frac{p(x)}{q(x)}-1)^2dx=\int_x \frac{(p(x)-q(x))^2}{q(x)}dx$$

2.Fenchel Conjugate (Fenchel共轭)

3. Connection with GAN