cGANs with Projection Discriminator

• paper: cGANs with Projection Discriminator

Introduction

• We propose a novel, projection based way to incorporate the conditional information into the discriminator of GANs that respects the role of the conditional information in the underlining probabilistic model (i.e., a function that measures the information theoretic distance between the generative distribution and the target distribution).
  • By construction, any assumption about the form of the distribution would act as a regularization on the choice of the discriminator. In this paper, we propose a specific form of the discriminator, a form motivated by a probabilistic model in which the distribution of the conditional variable $y$ given $x$ is discrete or uni-modal continuous distributions.
  • As we will explain in the next section, adhering to this assumption will give rise to a structure of the discriminator that requires us to take an inner product between the embedded condition vector $y$ and the feature vector (Figure 1d).
    

The Architecture of the cGAN Discriminator with a Probablistic Model Assumptions

• Ref: 理解 cGANs with Projection Discriminator

---

Notation

• $\mathbf{x}$: input vector
• $\mathbf{y}$: conditional information (When $\mathbf{y}$ is discrete label information, we can assume that it is encoded as a one-hot vector.)
• $D(\mathbf{x},\mathbf{y};\theta ):=\mathcal{A}(f(\mathbf{x},\mathbf{y};\theta ))$: cGAN discriminator, where $\mathcal{A}$ is an activation function
• $q$: the true distributions
• $p$: the generated distributions

$f^{\ast }(x,y)$

• The standard adversarial loss for the discriminator is given by:
  with $\mathcal{A}$ in $D$ representing the sigmoid function.
• 类似 GAN 中的推导，假设 $D$ 可以表示任意函数，则可以推导出 optimal discriinator $D^{\ast }(x,y)$:
  $$D^{\ast }(x,y)=\frac{q(x,y)}{q(x,y)+p(x,y)}$$由于现在假设激活函数为 sigmoid，因此有
  $$\mathcal{A}(f(x,y;\theta ))=\frac{1}{1+\exp (-f^{\ast }(x,y))}=D^{\ast }(x,y)=\frac{q(x,y)}{q(x,y)+p(x,y)}$$因此有
  $$f^{\ast }(\mathbf{x},\mathbf{y})=\log \frac{q(\mathbf{x},\mathbf{y})}{p(\mathbf{x},\mathbf{y})}=\log \frac{q(\mathbf{x}\mid \mathbf{y})q(\mathbf{y})}{p(\mathbf{x}\mid \mathbf{y})p(\mathbf{y})}=\log \frac{q(\mathbf{y}\mid \mathbf{x})}{p(\mathbf{y}\mid \mathbf{x})}+\log \frac{q(\mathbf{x})}{p(\mathbf{x})}:=r(\mathbf{y}\mid \mathbf{x})+r(\mathbf{x})$$

Motivation behind the Projection Discriminator

log linear model

• Log linear model is the most popular model for $p(y|x)$. Assume that $y$ is a categorical variable taking a value in { 1 , . . . , C } \{1, . . . , C\} {1,...,C}.
• 如果我们要 softmax 来计算 $x$ 属于各个类别的概率，则
  $$p(y=c|x)=\frac{\exp (o_c)}{{\sum }_{j=1}^C\exp (o_j)}$$其中 $o_j$ 为神经网络全连接层的输出，我们可以把它分解成全连接层的权重矩阵 $V^{pT}$ (size: $C\times d^L$) (上标 $p$ 代表该权重与真实概率 $p$ 有关) 与输入向量 $\varphi (x)$ (size: $d^L\times 1$) (代表提取出的 $x$ 的 feature) 的乘积。这样代表类别概率的输出向量 $o$ 就可以表示为 $o=V^{pT}\varphi (x)$，而其中 $o_j$ 为:
  $$o_j=v_j^{pT}\varphi (x)$$因此，
  $$\begin{array}{rl}\log p(y=c|x)& =\log \frac{\exp (v_c^{pT}\varphi (x))}{{\sum }_{j=1}^C\exp (v_j^{pT}\varphi (x))}\\ & =v_c^{pT}\varphi (x)-\log (\sum _{j=1}^C\exp (v_j^{pT}\varphi (x)))\end{array}$$设 $Z^p(\varphi (x)):={\sum }_{j=1}^C\exp (v_j^{pT}\varphi (x))$，则
  $$\begin{array}{rl}\log p(y=c|x)& =v_c^{pT}\varphi (x)-\log Z^p(\varphi (x))\end{array}$$
• 假设 $\log q(y=c|x)$ 也可以表示为上面的形式，并且使用同样的 $\varphi$，则下面的对数似然比可表示为
  $$\begin{array}{rl}\log \frac{q(y=c|x)}{p(y=c|x)}& =v_c^{qT}\varphi (x)-\log Z^q(\varphi (x))-v_c^{pT}\varphi (x)+\log Z^p(\varphi (x))\\ & =(v_c^q-v_c^p{)}^T\varphi (x)-(\log Z^q(\varphi (x))-\log Z^p(\varphi (x)))\end{array}$$

---

• 将 log linear model 代入 $f^{\ast }(x,y)$ 可得
  $$\begin{array}{rl}{f}^{\ast }(x,y=c)& =\log \frac{q(y=c\mid x)}{p(y=c\mid x)}+\log \frac{q(x)}{p(x)}\\ & =(v_c^q-v_c^p{)}^T\varphi (x)-(\log Z^q(\varphi (x))-\log Z^p(\varphi (x)))+\log \frac{q(x)}{p(x)}\end{array}$$
• 设 $v_c=v_c^q-v_c^p$; $\psi (\varphi (x))=-(\log Z^q(\varphi (x))-\log Z^p(\varphi (x)))+\log \frac{q(x)}{p(x)}$，则
  $$\begin{array}{rl}{f}^{\ast }(x,y=c)& =v_c^T\varphi (x)+\psi (\varphi (x))\end{array}$$
• 设 $V$ 的各行向量为 $v_c^T$. 因为 $y$ 为 one-hot vector，则
  $$f^{\ast }(x,y)=y^{T}V\varphi (x)+\psi (\varphi (x))=(V^{T}y)\cdot \varphi (x)+\psi (\varphi (x))$$得出了下图的结构 (左路可以看作是在判断 $x$ 是否真实，右路可以看作在判断 $x$ 是否属于 $y$ 类)：
  

We refer to this model of the discriminator as projection for short.