An Introduction to GAN

What is GAN

GAN(Generative Adversarial Nets) is an unsupervised learning algorithm of deep learning, first proposed by Ian J. Goodfellow in 2014. The main innovation of GAN is to integrate ideas of competition and confrontation with the principal of data generation. For implementation, GAN achieves its goal with the help of neural network.

As GAN is often used to generate images, when explaining the principal of GAN, we use ‘image’ instead of data.

The Composition of GAN

There are two models in GAN, namely the generative model and the discriminative model. Given an image (real image), while the goal of generative model is to create an image(fake image) the same as the given image, the role of the discriminative model is to distinguish the real image from the fake image. Therefore, the generative model will optimize its parameters to make the fake image more similar to the real image to prevent the discriminative model from successfully distinguishing. Then, the discriminative model also optimizes its parameters to have a higher discriminative ability. After a long time competition between the generative model and the discriminative model, parameters in the two models reached convergence when the discriminative model could not tell which image is real and which image is fake(this is explained in The Mathematical Foundation of GAN).

The principle of GAN can also be represented by the figure below:

                        Figure1 The principle of GAN

In this figure, $G$(generative model) produces images from random noise and $D$(discriminative model) distinguishes the real image from the fake image.

About Value(Loss) Function

Here, we define the value function for GAN and the output of discriminative model is in the interval of (0,1):

Based on the principal that if an imaged is distinguished as a real image, the output of discriminative model is 1 and if an imaged is distinguished as a fake image, the output is 0, we use cross entropy as the value function. During optimization, the discriminative model aims to maximize the value function while the goal of generative model is to do the opposite way. Thus, in order to maximize $V(D,G)$, the discriminative model needs to try its best to distinguish the real image from the fake image. To minimize $V(D,G)$, the generative model has to make the fake image more similar to the real image, which can be inferred from the formula.

We use the principal of gradient descent and gradient ascent to update parameters in neural networks. It should be noted that in early learning, when $G$ is poor, $D$ can reject fake images with high confidence because they are clearly different from real images. In this case,
$log(1-D(G(z)))$ saturates, which prevents updates of parameters in $G$. Rather than training $G$ to minimize $log(1-D(G(z)))$ , training $G$ to maximize $logD(G(z))$ might be a better choice. However, the author of GAN fails to give the mathematical proof of whether the value function will converge if urging the generative model to maximize $logD(G(z))$. Therfore, in algorithm of GAN, the goal of the generative model is still to minimize $log(1-D(G(z)))$.
The goal of two models are listed as below:

discriminative model:
$max$ $log(D(x))$+$log(1-D(G(z)))$

generative model:
$min$ $log(1-D(G(z)))$

The Mathematical Foundation of GAN

For images given to the discriminative model, they may come from real images. They may also come from the generative model. We can consider that real images have their probability distribution and images generated from noise have another distribution. Once the distribution of noises are given and parameters of the generative model are fixed, the probability distribution of fake images that are from noises are determined. For an image given to the discriminative model, we call it as x. Here, x has $p_{data}$(x) probability comes from real data and $p_g$(x) probability comes from fake data.

Now we are going to prove two conclusions. First, we are going to prove that this formula
can reach a global optimum for $p_g$($x$)=$p_{data}$$(x$). After that, we will show the algorithm of GAN and prove that this algorithm promotes $p_g$($x$) to converge to $p_{data}$($x$).

The Proof of First conclusion

To prove the first conclusion, we need to first prove that for any given generator G, the optimal discriminator D will achieve the following equation:

Proof:

$V(D,G)$$=$$E_{x\sim }$$p_{data}$$log(D(x))$ + $E_{x\sim }$$p_g$$log(1-D(x))$

                $=$${\int }_x$$p_{data}$($x$)$log(D(x))$$dx$$+$ ${\int }_x$$p_g$($x$)$log(1-D(x))$$dx$

                $=$${\int }_x$$p_{data}$($x$)$log(D(x))$+$p_g$($x$)$log(1-D(x))$$dx$

As the goal of D is to maximize this value function, according to the principle of second derivative and extreme points, D should optimize its parameters to reach the following equation:
in order to maximize $V(D,G)$.

Now we are going to prove that when D has achieved:

generator G would make the minimum of value function only if G has made $p_g$($x$)=$p_{data}$($x$).

To prove this, we should introduce JS divergence, a variant version of KL divergence, who solves the problem of asymmetry of KL divergence. The definition of JS divergence is:

$D_{JS}$($p$||$q$)$=$$\frac{1}{2}$$D_{KL}$($p$||$\frac{p+q}{2}$)$+$$\frac{1}{2}$$D_{KL}$($q$||$\frac{p+q}{2}$),

where
Here, we calculate JS divergence between $p_g$($x$) and $p_{data}$($x$) first.

The formula is from the Internet and $p_r$ above is just exactly $p_{data}$ and $L$($G$,$D$) above is just $V$($G$,$D$) here. It is surprising to find that when calculating JS divergence between $p_g$($x$) and $p_{data}$($x$), $V$($G$,$D$) appears!

Hence, $V$($G$,$D$) $=$ $2$$D_{JS}$($p_{data}$||$p_g$)$-$$log4$.

As JS divergence is always non-negative, to make the minimum of $V$($G$,$D$), $D_{JS}$($p_{data}$||$p_g$) should be zero. At that time, $p_g$($x$)=$p_{data}$($x$) and $V$($G$,$D$) $=$$-$$log4$. What should be noted is at that time, $D^{\ast }$(x)=$\frac{1}{2}$, showing the discriminative model would not be able to distinguish the real image from the fake image successfully.

The Proof of Second conclusion

Here is the algorithm of GAN.

If G and D have enough capacity, and at each step of Algorithm, the discriminator is allowed to reach its optimum given G, and $p_g$ is updated so as to improve the criterion

then $p_g$($x$) converges to $p_{data}$($x$).

Proof:

For this proof, I don’t understand it well. It would be appreciated if somebody can explain it for me.

DCGAN

DCGAN is an advanced version of GAN, who combines the principal of convolutional neural network with GAN. In addition, DCGAN uses LeakyReLU as activation function and creates the idea of transposed convolution, replacing fully collected neural network with convolutional neural network. Experiments from DCGAN shows that DCGAN has better training stability than ordinary GAN.

The figure below shows the principal of DCGAN.

Finally, as it is easy to implement DCGAN with TensorFlow, it would be unnecessary to go into details here. For more detailed information about implementation, you can visit https://tensorflow.google.cn/

However, DCGAN has not been able to improve GAN from a deep level. The emergence of WGAN has completely solved many problems of GAN, such as instability！