Mathematical Principle of GAN
Introduction
The potential advantage of deep learning lies in representing complicate probability densities of data with models of large scale hierarchical structure. Till now, the great success of deep learning is mostly in discriminative models that usually map high-dimensional sensible inputs to categorical labels. Training discriminative models benefits a lot from back-propagation algorithm, dropout and ReLU with well-defined differentials. In contrast to discriminative models, however, generative deep models perform inferiorly. This is much due to the intractability of optimising joint likelihood.
Based upon the observation above, Goodfellow proposed generative adversarial net (GAN). As the name implies, GAN consists of two network: generator and discriminator. The generator produces some fake data as real as those genuine data that are assumed to be drawn from some unknown distribution (Only God knows what), while the discriminator tries to improve itself in distinguishing the genuine from the fake. The whole process is like a gambling game.
This formulation is actually a general framework, since both the discriminator and the generator can be a deep model of any kind. For sake of simplicity, in this work, we only consider the multi-layer perceptron. Besides, the data generated by the generator are obtained by transforming a random noise. With this approach, the entire model can be trained using back-propagation and dropout without approximation inference and Markov chain.
Generative Adversarial Net
In this section, we will see GAN in detail and derive its optimisation objective.
As mentioned above, both generator and discriminator are MLPs for simplicity. Philosophically, the generator is expected to something that maps a random noise into a generative data; the discriminator takes input either genuine or fake and outputs the probability of being genuine (drawn from the unknown distribution that only God knows). This whole framework is shown as follows,
Suppose we have a training set of
m
samples,
Further, the log likelihood is obtained
According to Large Number Theorem, as m→∞ , the empirical loss is exploited to approximate the expectation loss,
Back to our aim: the whole process is tantamount to a gambling game. The generator produces some fake data as real as those genuine data that are assumed to be drawn from some unknown distribution (Only God knows what), while the discriminator tries to improve itself in distinguishing the genuine from the fake. Note that the log likelihood above is the objective with respect to the discriminator D(⋅) . Therefore, on one hand, we want to optimize the learnable parameters of the discriminator by maximizing the log likelihood; on the other, we want to optimize those of the generator by minimizing the log likelihood. Then, our optimization objective becomes
Theoretical Results
The generator
G(⋅)
implicitly defines a probability distribution with pdf
pg
. By “implicitly”, I mean
G
is a mapping from the noise
Firstly, given any generator
G
, we consider the optimal discriminator
Proposition 1. For a given
G
, the optimal discriminator is given by
Proof.
For the given
G
, we maximize the quantity
where,
χ
and
Ω
are the sample space of
x
and
The second term, by x=G(z) , becomes
This equality is not that straightforward. (Refer to the details in Appendix C: Change of Random Variable in Measure Theory)
Consequently,
Since
G
is fixed, the unknown generative distribution that God created is thus fixed. This yields the non-zero functions
This completes the proof. #
By this proposition, we further reformulate this min-max gambling game as minimizing the quantity C(G) , where
Now, by presenting Theorem 1, we answer the question raised at the beginning: whether
pg
will reach
pdata
, the pdf of the unknown distribution only God knows? Or, how much between them if not?
Theorem 1. The global optimal minimum of
C(G)
can be obtained at
pg=pdata
, and the value is
−log4
.
Proof.
The equality holds iff
pdata=pg=pdata+pg2
. (Refer to Appendix B)
This completes the proof. #
As derived above, the generator is able to duplicate the process of generating data by God.
Formally, we present the GAN training algorithm as below
Noticeably, the algorithm presented differs from our optimization objective. Our objective is to optimize wrt D to the end and then optimize wrt G. But, in practice, this will bring prohibitive computational cost and usually result in overfitting. Rather, the algorithm optimizes D for k steps and then optimizes G. This procedure is carried out iteratively till converge. In this way, as long as G changes sufficiently slow, D is always near the optimal solution. Naturally, again, we are quite curious about the quantitative difference between these two procedures. This can be answered by the following proposition.
We present the convergence of the algorithm presented without proof.
Proposition 2. Suppose
G
and
then, pg converges to pdata .
For a better understanding, we take a look at the following pedagogical epigraph.
The downmost horizontal line represents the range of the noise and x represents the data range. As is shown,
G(z)
maps the uniform distribution to a non-uniform distribution. The green curve represents the pdf of
x=G(z)
,
pg
; the black curve represents the pdf of the unknown distribution created by God,
pdata
; the blue curve is the decision plane of
D
. (a) Before convergence,
Appendix
A. K-L Divergence
In probability and information theory, K-L divergence, also called information gain, is a “measure” of the distance between two probability densities. Here its formal definition is presented, in discrete form and continuous form, respectively.
Given two probability densities of two discrete random variables,
P
and
Given two probability densities of two continuous random variables,
p
and
Mathematically, K-L divergence is not a measure, since it violates one of the maxima of measure, i.e. symmetry, that is,
DKL(p∥q)≠DKL(q∥p)
. The former place, e.g.
P
(or
There are a few important properties of K-L divergence:
(1) K-L divergence is well-defined if and only if for some
x
such that
(2) For some
x
such that
(3) DKL(p∥q)≥0 . The equality holds iff p=q .
Now we prove the third property.
This completes the proof.#
B. Variational Calculus
Variational calculus is a natural extension of differentiation.
Given a functional
F[y]:y(x)↦
, where
= or
, in parallel with Taylor expansion, functional expansion is defined as follows, for any
η(⋅)
,
C. Change of Random Variable in Measure Theory
Proof:
where x=G(z) .
First, we define a measure space
(Ω,,ℙ)
, where
Ω
is the sample space of
z
,
where ℙG is the distribution of x <script type="math/tex" id="MathJax-Element-108">x</script>.
This completes the proof.#