tf中的GAN原理及变种
1. GAN基本原理
- GAN基本结构:
- GAN基本思想:训练一个判决器,要求其判决准度够高。训练一个生成器,使其生成的数据的分布与训练判决器数据的分布尽可能接近,即使 p g ( x ) p_g(x) pg(x) p r ( x ) p_r(x) pr(x)尽可能相似
- GAN的损失函数: m i n G m a x D L ( D , G ) = E x ∼ p r ( x ) [ log D ( x ) ] + E z ∼ p z ( z ) [ log ( 1 − D ( G ( z ) ) ) ] min_{G}\ max_{D}L(D,G) = E_{x\sim p_{r}(x)}[\log D(x)]+E_{z\sim p_{z}(z)}[\log (1-D(G(z)))] minG maxDL(D,G)=Ex∼pr(x)[logD(x)]+Ez∼pz(z)[log(1−D(G(z)))] m i n G m a x D L ( D , G ) = E x ∼ p r ( x ) [ log D ( x ) ] + E x ∼ p g ( x ) [ log ( 1 − D ( x ) ) ] min_{G}\ max_{D}L(D,G) = E_{x\sim p_{r}(x)}[\log D(x)]+E_{x\sim p_{g}(x)}[\log (1-D(x))] minG maxDL(D,G)=Ex∼pr(x)[logD(x)]+Ex∼pg(x)[log(1−D(x))]
2. 纳什均衡
- GAN训练方法:一般在训练GAN时,首先固定G,训练D,然后再固定D,训练G
- KL散度: D K L ( p ∣ ∣ q ) = ∫ ∞ p ( x ) log p ( x ) q ( x ) d x D_{KL}(p\mid\mid q) = \int_\infty p(x)\log\frac {p(x)}{q(x)}dx DKL(p∣∣q)=∫∞p(x)logq(x)p(x)dx
- JS散度: D J S ( p ∣ ∣ q ) = 1 2 D K L ( p ∣ ∣ p + q 2 ) + 1 2 D K L ( q ∣ ∣ p + q 2 ) D_{JS}(p\mid\mid q) = \frac{1}{2}D_{KL}(p\mid\mid \frac{p+q}{2}) + \frac{1}{2}D_{KL}(q\mid\mid \frac{p+q}{2}) DJS(p∣∣q)=21DKL(p∣∣2p+q)+21DKL(q∣∣2p+q)
- Discriminator(D):GAN的损失(质量)函数为 V ( D , G ) = ∫ ∞ p r ( x ) log D ( x ) + p g ( x ) log ( 1 − D ( x ) ) d x V(D,G) = \int_\infty p_{r}(x)\log D(x)+p_{g}(x)\log (1-D(x))dx V(D,G)=∫∞pr(x)logD(x)+pg(x)log(1−D(x))dx在固定G后,可以简化为 D G ∗ ( x ) = p r ( x ) p r ( x ) + p g ( x ) D_G^*(x) = \frac{p_r(x)}{p_r(x)+p_g(x)} DG∗(x)=pr(x)+pg(x)pr(x)
- Generator(G): D J S ( p r ∣ ∣ p g ) = log 4 + L ( G , D ∗ ) 2 D_{JS}(p_r\mid\mid p_g) = \frac{\log 4 + L(G,D^*)}{2} DJS(pr∣∣pg)=2log4+L(G,D∗) L ( G , D ∗ ) = 2 D J S ( p r ∣ ∣ p g ) − 2 log 2 L(G,D^*) = 2D_{JS}(p_r\mid\mid p_g)-2\log2 L(G,D∗)=2DJS(pr∣∣pg)−2log2
3. DCGAN
- DCGAN结构示意:
- Transposed Convolution:一般情况下,卷积用于降维。而Transposed Convolution通过padding的方式来增维
- DCGAN需要精细设计参数,才可得到较好的结果
4. EM距离
- JS和KL散度存在的问题:KL和JS散度在p和q相差较大的时候,更新十分困难(梯度弥散)。对于GAN,初始状态(一般为高斯噪声)与所需生成的数据之间的差距一般差别很大,这就导致了GAN的初期训练十分艰难
- Wasserstein距离:Wasserstein距离指把一个分布转化为另一个分布的最小步数
- 【数学】Wasserstein Distance
5. WGAN
- 相比于传统GAN的损失函数(基于JS散度),WGAN采用Wasserstein Distance作为损失函数,有效地解决了初期训练困难的问题
- WGAN采用Wasserstein Distance作为损失函数,需要满足下式(1-Lipschitz Function) ∣ f ( x 1 ) − f ( x 2 ) ∣ ≤ ∣ x 1 − x 2 ∣ \mid f(x_1)-f(x_2)\mid \leq \mid x_1-x_2\mid ∣f(x1)−f(x2)∣≤∣x1−x2∣为了满足此式采用Weight Clipping,即把权值限定在[-a,a]中,若w>a,则w=a;若w<-a,则w=-a;Weight Clipping思想即把参数总量限制在一定范围内
- WGAN的损失函数: L = E x ˇ ∼ P g [ D ( x ˇ ) ] − E x ∼ P r [ D ( x ) ] + λ E x ^ ∼ P x ^ [ ( ∣ ∣ ∇ x ^ D ( x ^ ) ∣ ∣ 2 − 1 ) 2 ] L = E_{\check{x}\sim P_g}[D(\check{x})] - E_{x\sim P_r}[D(x)] + \lambda E_{\hat{x}\sim P_{ \hat{x} }}[(\mid\mid \nabla_{\hat{x}}D(\hat{x}) \mid\mid_2 - 1)^2] L=Exˇ∼Pg[D(xˇ)]−Ex∼Pr[D(x)]+λEx^∼Px^[(∣∣∇x^D(x^)∣∣2−1)2] x ^ = t x ˇ + ( 1 − t ) x 0 ≤ t ≤ 1 \hat{x} = t\check{x} + (1-t)x \ \ \ \ \ \ \ \ \ \ \ \ \ 0 \leq t \leq 1 x^=txˇ+(1−t)x 0≤t≤1其中前两部分是计算损失,最后一个部分是限制参数(梯度)在某一范围
- WGAN的初期训练不需要精细设计参数,就可得到较好的结果