学习论文 "Zero-shot Learning via Simultaneous Generating and Learning"笔记

Notation

$\theta$:represents the parameters of the deep neural network that define the decoder.
$\psi$:represents the parameters of the deep neural network that define the encoder.
$q_{\varphi }(z|x,y)$:traditional encoder,作为目标识别模型，need y for both training and testing

如果对概率图模型的随机变量的向量$X$,$Z$进行建模得到的生成模型$p_{\theta }(x)={\sum }_z{p}_{\theta }(z)p_{\theta }(x|z).$一般此求和式不可以写成有限运算形式，故无法通过ML算法优化，同时第二项的似然$p_{\theta }(z|x)=\frac{p_{\theta }(x,z)}{p_{\theta }(z)}$中含有的prior也不可计算（p(z)是隐变量的概率，这是我要采样转化的，但是目前seen很少，无法得到转化的概率分布，不然还求啥p(x)）
所以用AEVB算法，$q_{\varphi }(z|x,y)$近似似然$p_{theta}(z|x)$同时把最大化似然函数下界作为目标函数的方法，巧妙地避开了后验概率计算。（听不懂没关系，禁止套娃记结论）

$p_{\theta }(x|z,y)$:traditional decoder, 是隐变量的后验分布，need y for both training and testing

Cause $y$, when labels missing, both encoder and decoder are hardly
exploited and decoder only takes advantage when generating datapoints
in certain condition

If adopting assume: $p(z|y)\to p(z)$,exploiting the latent variables for classification becomes new challenge.

$A^{\ast }$:class attribute embedding vectors
$p(X^{s\ast }|A_y^{s\ast })$将类别用类嵌入向量$A\ast$表示，通过中间隐变量$z$生成$x$的过程
$p_{\psi }(z|a)$:attribute-specific gaussian distribution of z.

$\mu (x^{s\ast })$:the mean of the approximated variational likelihood $q_{\varphi }(z|x^{s\ast })$
$p(x|a_y^{u\ast })\simeq {\int }_z{p}_{\hat{\theta }}(x|z)p_{\hat{\psi }}(z|a_y^{u\ast })$:GENERATIVE MODEL,generate unseen sampled datapoints for [18]
$\hat{\Theta }$:generative model parameters need optimized.
$x^I$ 辅助变量

Abbreviation

SGAL: Simultaneously Generating And Learning

Motivation

1.Problem

Traditional models are trained mainly on the seen classes. Obtaining the generative model for
both seen and unseen is quite far from their consideration, since scarcity of the unseen samples is apparently a fundamental problem for ZSL.

2.Existing Solution and Difficulty

• CVAE: generate conditional samples
• GAN
  Difficulty: Need training additional off-the-shell classifier

Those methods based on deep generative model train their models first and generate enough samples for unseen classes and subsequently train additional classifier, rather than training conditional generative models for both seen and unseen classes.

Contribution of this paper

• 数据问题：为了推理大量含有missing label $y$的数据，无中生有，逼近其真实的标签，传统VAE面临unknown intent identify的问题，Assuming that categories represented as class embedding vectors $a^{\ast }$,模型学习attribute-specific的潜在空间分布，再通过隐变量生成$x$（曲线救国）
• 任务驱动：since our purpose is classification（not just generative）,assuming the conditional prior to be a category-specific Gaussian distribution ，所以引入最大化间隔准则来促使$q_{\varphi }(z|x^{s\ast })$与其他类别的 $p_{\psi }(z|a_{y^{s\ast }}^{\ast })$越远越好.(Trick: Margin Regularizer)
• 训练过程：treat these missing datapoints as variables that should be optimized as well as
  model parameters. 把这些missing的数据点对应分布的参数当做变量（EM参数估计），作为整个模型参数优化的一部分，随生成模型训练参数优化更新而一并更新（个人感觉是这篇文章的重点工作）

Methodology

1. 给出了class semantic(or attribute) embedding vectors A ∗ = { a k ∗ } k = 1 S + U A*=\{a^*_k\}^{S+U}_{k=1} A∗={ak∗​}k=1S+U​ including both seen and unseen的属性向量，为attribute-specific方法做准备。

1 Category-Specific Multi-Modal Prior and Classification

2. Input variable independently: the latent prior $p(z|y)\to p(z)$, and exploiting the latent var $z$ for classification
3. VAE可以通过隐变量学习数据的复杂模型。不同于标准VAE的隐变量z的先验分布是服从标准正态分布，我们假设每个zn是从attribute-specific的高斯分布 $p_{\psi }(z|a_{y^{s\ast }}^{\ast })$得到的

Inductive ZSL

Standard VAE prior: 隐变量$z_n$从高斯分布$p_{\psi }(z_n)=N(0,I)$采样得到。
attribute-specific VAE prior:隐变量的先验从高斯分布$p_{\psi }(z_n|a_n)=N(\mu (a_n),\sum (a_n))$采样得到。

根据Inductive ZSL method,考虑seen class$X^{s\ast }$和对应的类别向量标签$A_y^{s\ast }$的概率关系已知用边界似然。目的是建立隐变量$z$与seen数据的属性向量之间的映射关系，从而解决了大量missing label $y$情况下无法通过现有real $x$生成$z$的问题。其VAE下边界变成：$$L(\Theta ;x^{s\ast },a_{y_i^{s\ast }}^{\ast })=-KL(q_{\varphi }(z|x^{s\ast }||p_{\psi }(z|a_{y_i^{s\ast }}^{\ast }))+E_{z\sim q}[log{p}_{\theta }(x^{s\ast }|z)]$$
既然目标是先生成$z$，那么就需要建立$z$与$a_{y_i^{s\ast }}^{\ast }$的映射。继续Assuming that条件先验概率$p_{\psi }(z|a_{y_i^{s\ast }}^{\ast })$是category-specific Gaussian distribution的，那么$a_{y_i^{s\ast }}^{\ast }$就是一个样本空间的可数无限的划分，满足全概率公式$P(A)={\sum }_{n}P(A|B_n)P(B_n)$，因此就可以计算隐变量的概率分布：$$p(z)=\sum _{i}p(a_{y_i^{s\ast }}^{\ast })p_{\psi }(z|a_{y_i^{s\ast }}^{\ast })$$
得到$z$的先验目的是为了分类，而如何能够让不同先验值的$z$更加简单清晰地反映不同类别呢？
作者采用属性向量归一化和先验正则化Loss的方法promotes each cluster of $p_{\psi }(z|a_{y_i^{s\ast }}^{\ast })$to be far away from all other clusters above the certain distance in latent space $z$
再通过KL散度$(q_{\varphi }(z|x^{s\ast }||p_{\psi }(z|a_{y_i^{s\ast }}^{\ast }))$得到的encoder$q_{\varphi }(z|x^{s\ast })$就可以逼近不可解的条件似然$p_{\psi }(z|a_{y_i^{s\ast }}^{\ast })$，这是VAE的经典之处，即经典的intractability 问题

intractability:
$p_{\theta }=\int p_{\theta }(z)p_{\theta }(x|z)dz$,公式不可以写成有限运算的形式，则不可计算，故无法通过ML算法进行参数优化。同时$p_{\theta }(z|x)=p_{\theta }(z,x)/p_{\theta }(x)$中含有先验概率也不可计算，故无法使用EM算法对参数进行优化以及进行所有基于后验概率的推理。为了解决这个问题，VAE采用$q_{\varphi }(z|x)$近似$p_{\theta }(z|x)$ 同时把最大化似然函数下界作为目标函数的方法，巧妙地避开了后验概率计算。这一过程数学工具与模型与上文一致。

最后回到优化问题上，利用MLE方法对label $\hat{y}$估计：$$\hat{y}=argma{x}_{y^{s\ast }}p(x^{s\ast }|a_{y_i^{s\ast }}^{\ast })\simeq argma{x}_{y^{s\ast }}{p}_{\psi }(z=\mu (x^{s\ast })|a_{y^{s\ast }}^{\ast })$$
其中$\mu (x^{s\ast })$就是上式边界的先验似然逼近函数$q_{\varphi }(z|x^{s\ast })$的均值。

By simply calculating Euclidian distances between category-specific multi-modal and the encoded variable $\mu (x^{s\ast })$,classification results can be achieved.（不同类别转化成不同参数的高斯分布，通过分布的均值区别不同的类别（你用方差也行））

We train the model using the seen-class labeled examples and learn the parameters $\hat{\Theta }$ by maximizing the objective in $L(\Theta ;x^{s\ast },a_{y_i^{s\ast }}^{\ast })$.Once the model parameters have been learned, the label for a new input $\hat{x}$ from an unseen class can be predicted by first predicting its latent embedding $\hat{z}$ using the VAE recognition model, and then finding the “best” label by solving$$\hat{y}=argma{x}_{y^u\ast }L(\Theta ,\hat{x},A_y^{u\ast })=argma{x}_{y^{u\ast }}p(\hat{x}|a_{y_i^{u\ast }}^{\ast })\simeq argma{x}_{y^{u\ast }}{p}_{\psi }(z=\mu (x^{u\ast })|a_{y^{u\ast }}^{\ast })$$
就可以生成unseen的$\hat{x}$的$\hat{y}$（ the model is trained on seen classes $A^{s\ast }$, we can try to use the generative model by simply inputting the embedding vector of unseen classes ${}^{u\ast }$）

2 Generative Model for both Seen and Unseen Classes

Focus: training process, the absence of datapoints Xu∗ for unseen classes is a fundamental problem
in ZSL.
So treat these missing datapoints as variables that should be optimized as well as model parameters.
反向思维，假设lower bound已经完美的逼近了我们的target distribution for both seen and unseen class，边界参数$\Theta$和unseen datapoint $X^{u\ast }$会同时满足：

这是生成模型常见的一种递归学习方法。从生成器
$p_{\hat{\theta }}(x|z)p_{\hat{\psi }}(z|a_y^{u\ast })$采样得到missing datapoint $X^{u\ast }$，在用这个采样的batch和seen existing datapoints$X^{s\ast }$一起喂回去优化生成器参数$\hat{\Theta }=argma{x}_{\Theta }{\sum }_{k=1}^S\frac{1}{N}{\sum }_{x_n\sim p(x|a_k^{s\ast })}^{N}logp(x_n|a_k^{s\ast };\Theta )$。并通过SGAL递归求解上面两式完成模型参数训练。但是注意啊，我们采的missing datapoint 不是瞎采样的，而是需要采样的$X^{u\ast }$近似服从我们的unseen classes $p(x|a^{u\ast })$这样一个target likelihood.
但是我们并不知道target likelihood是什么…（如果我们是有监督学习，我们就知道目标分布是什么样子，但是现在unseen烦就烦在不知道这货的分布是什么）

所以这里需要EM算法，既然我不知道精确分布，我估计一个大致分布也可以啊。

其中$x$和$x^I$are assumed to be a feature vector and its corresponding image,We assume that generating $x$ is only affected by $a$, and $x^I$ is depend only on $x$.

使用EM算法的目的，就是要把这货分布的参数$\Theta$给估计出来,那就对对数似然函数求导嘛：

对比一下$$\hat{\Theta }=argma{x}_{\Theta }\sum _{k=1}^S\frac{1}{N}\sum _{x_n\sim p(x|a_k^{s\ast })}^{N}logp(x_n|a_k^{s\ast };\Theta )$$和$$argma{x}_{\Theta }\sum _{k=S+1}^{S+U}\frac{1}{N}\sum _{x_n\sim p(x|a_k^{u\ast };{\Theta }^{old})}^{N}logp(x_n|a_k^{u\ast };\Theta )$$

采用EM算法的参数优化方法，与传统方法的区别在：

1. previous model采样$x_n\sim p(x|a_k^{u\ast };{\Theta }^{old})$过程，将unseen的属性向量作为了先验知识（与上文unseen + existing labeled datapoint对应），作为算法的E-step；
2. VAE方法逼近真实分布的参数$\Theta$,作为算法的M-step:$$L(\Theta ;x^{s\ast },a_{y_i^{s\ast }}^{\ast })=-KL(q_{\varphi }(z|x^{s\ast }||p_{\psi }(z|a_{y_i^{s\ast }}^{\ast }))+E_{z\sim q}[log{p}_{\theta }(x^{s\ast }|z)]$$

算法的structure，感觉其实还不如直接看算法流程

但是在EM算法迭代优化$\Theta$的过程中，由于存在$X^u$ is generated from the incomplete model which is still in the training process(不确定性问题), thus handle the uncertainty by assuming the model parameters to be Bayesian random variables. (也就是说对生成器模型$p_{\hat{\theta }}(x|z)p_{\hat{\psi }}(z|a_y^{u\ast })$进行处理）

生成器中的unseen classes $p_{\hat{\psi }}(z|a_y^{u\ast })$可以展开为：

The model uncertainty can be considered by activating dropouts in each network.

3 More Conclusion of the structure

2.13号又继续看了一下作者的代码部分，感觉前面对文章中prior network理解的不够彻底，没有提及这一关键的structure,正如作者的注释：

#==================== VAE structure example =========================
# vaeStructure = {
#     'name' : 'VAE',
#     'encoder': {
#         'name' : 'encoder',
#         'trainable' : True,
#         'inputDim' : 2048,
#         'hiddenOutputDimList' : [512, 128],
#         'outputDim' : 128*2,
#         'hiddenActivation' : tf.nn.leaky_relu,
#         'lastLayerActivation' : None,
#     },
#
#     'decoder': {
#         'name' : 'decoder',
#         'trainable' : True,
#         'inputDim' : 128,
#         'hiddenOutputDimList' : [512],
#         'outputDim' : 2048,
#         'hiddenActivation' : tf.nn.leaky_relu,
#         'lastLayerActivation' : None,
#     },
#
#     'priorNet': {
#         'name' : 'priorNet',
#         'trainable' : True,
#         'inputDim' : 85,
#         'hiddenLayerNum' : 2,
#         'outputDim' : 128,
#         'hiddenActivation' : tf.nn.leaky_relu,
#         'lastLayerActivation' : None,
#         'constLogVar' : None,
#     },
# }

priorNet是针对采样sampling部分的一个先验网络，在上文中的EM部分的M step( Step 2)对应着$E_{\varphi }$，为什么采用先验网络也是为了方便cluster的一个操作，具体可以参考参考文献32，讲得十分详细。
Encoder和Decoder部分的代码基本和传统VAE理解思路是一样的。

Appendix

Margin Regularizer

The objective in $L$ naturally encourages the inferred variational distribution$q_{\varphi }(z|x)$to be close to the class-specific latent space distribution$p_{\theta }(z|x)$. However, since our goal is classification, we augment this objective with a maximum-margin criterion that promotes $q_{\varphi }(z|x)$ to be as far away as possible from all other class-specific latent space distributions.