Learning in Implicit Generative Models

对于隐生成模型来说，其直接定义了生成过程，如GAN中的生成器，没有似然函数，对于这一类模型的学习，就不能如VAE那样通过最大化似然函数得到。那么可以基于这样一个假设:真实的数据分布跟所定义的生成模型的分布相等$p^{\star }(\mathbf{x})=q_{\theta }(\mathbf{x})$。主要通过两个步骤进行学习：比较和估计。对于比较步骤：利用density difference $r(\mathbf{x})=p^{\star }(\mathbf{x})-q_{\theta }(\mathbf{x})$或者density ratio $r(\mathbf{x})=p^{\star }(\mathbf{x})/q_{\theta }(\mathbf{x})$，利用比较器$r(\mathbf{x})$能够区分模型生成的数据与真实数据的相差程度。对于估计步骤：利用比较器所能提供的信息进而更新隐生成模型的参数$\theta$。
总共有四种方法进行隐模型的学习，如图所示。

Class Probability Estimation

设数据为$\mathcal{X}\subset {\mathbb{R}}^d$，从真实数据分布中得到$n$个样本 X p = { x 1 ( p ) , … , x n ( p ) } \mathcal X_p=\{\mathbf x_1^{(p)},\dots,\mathbf x_n^{(p)}\} Xp​={x1(p)​,…,xn(p)​}，同样地，从模型分布中得到$n^{\prime }$个样本 X q = { x 1 ( q ) , … , x n ′ ( q ) } \mathcal X_q=\{\mathbf x_1^{(q)},\dots,\mathbf x_{n^\prime}^{(q)}\} Xq​={x1(q)​,…,xn′(q)​}。除此之外，对属于真实分布的样本赋予$y=1$，而对于模型分布的样本赋予$y=0$。这样一来我们可以表示$p^{\star }(\mathbf{x})=p(\mathbf{x}|y=1),q_{\theta }(\mathbf{x})=p(\mathbf{x}|y=0)$，则$$\begin{array}{rl}\frac{p^{\star }(\mathbf{x})}{q_{\theta }(\mathbf{x})}& =\frac{p(\mathbf{x}|y=1)}{p(\mathbf{x}|y=0)}=\frac{p(y=1|\mathbf{x})p(\mathbf{x})}{p(y=1)}/\frac{p(y=0|\mathbf{x})p(\mathbf{x})}{p(y=0)}\\ & =\frac{p(y=1|\mathbf{x})}{p(y=0|\mathbf{x})}\frac{1-\pi }{\pi }\end{array}$$可以发现ratio估计实则在估计类概率。其中$p(y=1)=\pi$，代表类边缘分布，一般人为设定，常常定义为$\pi =1/2$，或者对于不平衡的数据可以定义$\frac{1-\pi }{\pi }\approx n^{\prime }/n$。
下面我们的任务变为了指定一个配分函数或者判别器$\mathcal{D}(\mathbf{x};\mathbf{\varphi })=p(\mathbf{y}=1|\mathbf{x})\in [0,1]$。则密度比与判别器结果的关系为$\mathcal{D}=r/(r+1);r=\mathcal{D}/(1-\mathcal{D})$。常见的配分函数有
一般来说选择Bernoulli loss$$\begin{array}{l}\mathcal{L}(\mathbf{\varphi },\mathbf{\theta })\\ ={\mathbb{E}}_{p(\mathbf{x}|y)p(y)}[-y\log \mathcal{D}(\mathbf{x};\mathbf{\varphi })-(1-y)\log (1-\mathcal{D}(\mathbf{x};\mathbf{\varphi }))]\\ =\pi {\mathbb{E}}_{p^{\ast }(\mathbf{x})}[-\log \mathcal{D}(\mathbf{x};\mathbf{\varphi })]+(1-\pi ){\mathbb{E}}_{q_{\theta }(\mathbf{x})}[-\log (1-\mathcal{D}(\mathbf{x};\mathbf{\varphi }))]\end{array}$$由于$q_{\theta }(\mathbf{x})$为生成器，则$$\begin{array}{rl}\mathcal{L}(\mathbf{\varphi },\mathbf{\theta })& =\pi {\mathbb{E}}_{p\ast (\mathbf{x})}[-\log \mathcal{D}(\mathbf{x};\mathbf{\varphi })]\\ & +(1-\pi ){\mathbb{E}}_{q(\mathbf{z})}[-\log (1-\mathcal{D}(\mathcal{G}(\mathbf{z};\mathbf{\theta });\mathbf{\varphi }))]\end{array}$$以上结果刚好为GAN所使用的目标函数。优化上式可以两次优化(bi-level optimisation)：$$\begin{array}{l}\text{ Ratio loss: }{\min }_{\varphi }\pi {\mathbb{E}}_{p^{\ast }(\mathbf{x})}[-\log \mathcal{D}(\mathbf{x};\varphi )]+(1-\pi ){\mathbb{E}}_{q_{\theta }(\mathbf{x})}[-\log (1-\mathcal{D}(\mathbf{x};\varphi ))]\\ \text{ Generative loss: }{\min }_e{\mathbb{E}}_{q(\mathbf{z})}[\log (1-\mathcal{D}(\mathcal{G}(\mathbf{z};\mathbf{\theta })))]\end{array}$$

Divergence Minimisation

第二个方法是计算$p^{\star }$和$q$之间的散度。这就不得不提到f-散度。$$\begin{array}{c}{D}_f\left[p^{\ast }(\mathbf{x})\Vert q_{\theta }(\mathbf{x})\right]=\int q_{\theta }(\mathbf{x})f\left(\frac{p^{\ast }(\mathbf{x})}{q_{\theta }(\mathbf{x})}\right)d\mathbf{x}\\ ={\mathbb{E}}_{q_{\theta }(\mathbf{x})}[f(r(\mathbf{x}))]\\ \ge {\sup }_t{\mathbb{E}}_{p^{\ast }(\mathbf{x})}[t(\mathbf{x})]-{\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[f^{†}(t(\mathbf{x}))\right]\end{array}$$其中$f$为凸函数，Fenchel conjugate为$f^{†}$（前提是$f$为凸函数，lower-semicontinuous函数）， f † ( t ) = sup ⁡ u ∈ d o m f { u t − f ( u ) } f^{\dagger}(t)=\sup _{u \in \mathrm{dom}_{f}}\{u t-f(u)\} f†(t)=u∈domf​sup​{ut−f(u)}这个共轭函数刚好有$f$的性质，它也同样有Fenchel conjugate，即$$f^{††}=f$$则$$\begin{array}{rl}{D}_f(P\Vert Q)& ={\int }_{\mathcal{X}}q(x)\underset{t\in {\mathrm{dom}}_{f^{†}}}{\sup }\left\{t\frac{p(x)}{q(x)}-f^{†}(t)\right\}\mathrm{d}x\\ & \ge \underset{t\in \mathcal{T}}{\sup }\left({\int }_{\mathcal{X}}p(x)t(x)\mathrm{d}x-{\int }_{\mathcal{X}}q(x)f^{†}(t(x))\mathrm{d}x\right)\\ & =\underset{t\in \mathcal{T}}{\sup }\left({\mathbb{E}}_{x\sim P}[t(x)]-{\mathbb{E}}_{x\sim Q}\left[f^{†}(t(x))\right]\right)\end{array}$$这样就能进行min-max的训练了。对于以上不等式取等时为$t^{\ast }(\mathbf{x})=f^{\prime }(r(\mathbf{x}))$，将其代入原式子得$$\mathcal{L}={\mathbb{E}}_{p^{\ast }(\mathbf{x})}\left[-f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)\right]+{\mathbb{E}}_{q_{\theta }(\mathbf{x})}[f^{†}\left(f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)\right]$$其中$r_{\varphi }=r^{\ast }=p^{\ast }/q_{\theta }$，则优化目标为$$\begin{array}{l}\text{ Ratio loss: }{\min }_{\varphi }{\mathbb{E}}_{p^{\ast }(\mathbf{x})}\left[-f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)\right]+{\mathbb{E}}_{q_{\theta }(\mathbf{x})}[f^{†}\left(f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)\right]\\ \text{ Generative loss: }{\min }_{\theta }{\mathbb{E}}_{q(\mathbf{z})}[-f^{†}\left(f^{\prime }(r(\mathcal{G}(\mathbf{z};\mathbf{\theta })))\right]\end{array}$$密度比暗示着$p^{\ast }(\mathbf{x})\approx \tilde{p}=r_{\varphi }(\mathbf{x})q_{\theta }(\mathbf{x})$，那么$$\begin{array}{rl}{D}_{KL}\left[p^{\ast }(\mathbf{x})\Vert \tilde{p}(\mathbf{x})\right]=& \int p^{\ast }(\mathbf{x})\log \frac{p^{\ast }(\mathbf{x})}{r_{\varphi }(\mathbf{x})q_{\theta }(\mathbf{x})}d\mathbf{x}+\int \left(r_{\varphi }(\mathbf{x})q_{\theta }(\mathbf{x})-p^{\ast }(\mathbf{x})\right)d\mathbf{x}\end{array}$$这是针对于非归一化的分布的KL散度。从而有$$\begin{array}{rl}\mathcal{L}=& {\mathbb{E}}_{p^{\ast }(\mathbf{x})}\left[-\log r_{\varphi }(\mathbf{x})\right]+{\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[r_{\varphi }(\mathbf{x})-1\right]-{\mathbb{E}}_{p^{\ast }(\mathbf{x})}\left[\log q_{\theta }(\mathbf{x})\right]+{\mathbb{E}}_{p^{\ast }(\mathbf{x})}\left[\log p^{\ast }(\mathbf{x})\right]\end{array}$$则很容易得到ratio loss，即关于$\varphi$的有关项。但是通过这个没法得到generative loss，由于第三项需要$\log q_{\theta }(\mathbf{x})$，这对于隐模型来说是无法得到的。

Ratio matching

直接优化真实密度比$r^{\ast }(\mathbf{x})=p^{\ast }(\mathbf{x})/q_{\theta }(\mathbf{x})$和估计密度比$r_{\varphi }(\mathbf{x})$，$$\begin{array}{rl}\mathcal{L}& =\frac{1}{2}\int q_{\theta }(\mathbf{x}){\left(r(\mathbf{x})-r^{\ast }(\mathbf{x})\right)}^{2}d\mathbf{x}\\ & =\frac{1}{2}{\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[r_{\varphi }(\mathbf{x}{)}^2\right]-{\mathbb{E}}_{p^{\ast }(\mathbf{x})}\left[r_{\varphi }(\mathbf{x})\right]+\frac{1}{2}{\mathbb{E}}_{p^{\ast }(\mathbf{x})}\left[r^{\ast }(\mathbf{x})\right]\\ & =\frac{1}{2}{\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[r_{\varphi }(\mathbf{x}{)}^2\right]-{\mathbb{E}}_{p^{\ast }(\mathbf{x})}\left[r_{\varphi }(\mathbf{x})\right]\text{ s.t. }{r}_{\varphi }(\mathbf{x})\ge 0\end{array}$$利用上式可以很容易得到ratio loss和generative loss，进而进行优化。除了使用这种平方误差，还可以考虑使用Bregman divergence（以上的均方误差为其一个特例）$$\begin{array}{l}{B}_f\left(r^{\ast }(\mathbf{x})\Vert r_{\varphi }(\mathbf{x})\right)\\ ={\mathbb{E}}_{q_{\theta }(\mathbf{x})}(f\left(r^{\ast }(\mathbf{x})\right)-f\left(r_{\varphi }(\mathbf{x})\right)-f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)\left[r^{\ast }(\mathbf{x})-r_{\varphi }(\mathbf{x})\right])\\ ={\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[r_{\varphi }(\mathbf{x})f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)-f\left(r_{\varphi }(\mathbf{x})\right)\right]-{\mathbb{E}}_{p^{\ast }}\left[f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)\right]+D_f\left[p^{\ast }(\mathbf{x})\Vert q_{\theta }(\mathbf{x})\right]\\ ={\mathcal{L}}_B\left(r_{\varphi }(\mathbf{x})\right)+D_f\left[p^{\ast }(\mathbf{x})\Vert q_{\theta }(\mathbf{x})\right]\end{array}$$从上式可以得到ratio loss，即为${\mathcal{L}}_B\left(r_{\varphi }(\mathbf{x})\right)$,$$\begin{array}{l}{\mathcal{L}}_B\left(r_{\varphi }(\mathbf{x})\right)\\ =E_{p^{\ast }}\left[-f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)\right]+{\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[r_{\varphi }(\mathbf{x})f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)-f\left(r_{\varphi }(\mathbf{x})\right)\right]\\ =E_{p^{\ast }}\left[-f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)\right]+{\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[f^{†}\left(f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)\right)\right]\end{array}$$发现利用$$f^{†}\left(f^{\prime }(x)\right)=\underset{r}{\max }r{f}^{\prime }(x)-f(r)$$后，变为了上一节得到过的目标函数。进一步考虑，仍然提出关于$\theta$的部分，得到generative loss$$\begin{array}{rl}\mathcal{L}\left(q_{\theta }\right)=& {\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[r_{\varphi }(\mathbf{x})f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)\right]-{\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[f\left(r_{\varphi }(\mathbf{x})\right)\right]+D_f\left[p^{\ast }(\mathbf{x})\Vert q_{\theta }(\mathbf{x})\right]\end{array}$$发现其中仍然包含$q_{\theta }(\mathbf{x})$，无法继续求解。我们采用$p^{\ast }\approx r_{\varphi }{q}_{\theta }$，则$$\begin{array}{l}{D}_f\left[p^{\ast }(\mathbf{x})\Vert q_{\theta }(\mathbf{x})\right]={\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[f\left(\frac{p^{\ast }}{q_{\theta }(\mathbf{x})}\right)\right]\\ \approx {\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[f\left(\frac{q_{\theta }(\mathbf{x})r_{\varphi }(\mathbf{x})}{q_{\theta }(\mathbf{x})}\right)\right]={\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[f\left(r_{\varphi }(\mathbf{x})\right)\right]\end{array}$$这样一来可以得到$$\begin{array}{l}\text{ Ratio loss: }{\min }_{\varphi }{\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[r_{\varphi }(\mathbf{x})f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)-f\left(r_{\varphi }(\mathbf{x})\right)\right]-{\mathbb{E}}_{p^{\ast }}\left[f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)\right]\\ \text{ Generative loss: }{\min }_{\theta }{\mathbb{E}}_{q_{\theta }(\mathbf{x})}\left[r_{\varphi }(\mathbf{x})f^{\prime }\left(r_{\varphi }(\mathbf{x})\right)\right]\end{array}$$

Moment Matching

最后一个方法是检验$p^{\star }$和$q$的矩是否相同。$$\begin{array}{rl}\mathcal{L}(\varphi ,\mathbf{\theta })& ={\left({\mathbb{E}}_{p^{\ast }(\mathbf{x})}[s(\mathbf{x})]-{\mathbb{E}}_{q_{\theta }(\mathbf{x})}[s(\mathbf{x})]\right)}^2\\ & ={\left({\mathbb{E}}_{p^{\ast }(\mathbf{x})}[s(\mathbf{x})]-{\mathbb{E}}_{q(\mathbf{z})}[s(\mathcal{G}(\mathbf{z};\mathbf{\theta }))]\right)}^2\end{array}$$其中$s(\mathbf{x})$为某种统计量，其选择极其重要，一般来说我们希望所有矩都相同。