2019-8-22 paper reading

Ⅰ 3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image¹

1 Task

3D reconstruction from single view images.

2 Method

• Stage 1: train a point cloud auto-encoder $(E_p,D_p)$ to learn a latent space $\mathcal{Z}\in {\mathcal{R}}^k$ of 3D point clouds.
  – Loss function : Chamfer distance
• Stage 2: train an image encoder $E_I$ to map the 2D images to this learnt latent space $\mathcal{Z}$.
  ($b$): Match vectors in the latent $\mathcal{Z}$ space. 2D 图像的latent vector应该和对应的3D点云的latent vector相同. 以便在inference阶段根据$z_I$decode出来3D点云${\hat{X}}_I$.
  – latent matching loss: ${\mathcal{L}}_1(z_I-z_p)=|z_I-z_p|$ or ${\mathcal{L}}_2(z_I-z_p)=|z_I-z_p|$
  ($c$): Generate multiple plausible outputs, Learn a probabilistic distribution in the latent space.
  – Reparameterization trick
  – Formulate the latent representation $z_1$ of a specific input image $I_1$ to be a Gaussian random variable, i.e. $z_1\sim \mathcal{N}(\mu ,{\sigma }^2)$. The image encoder predicts the mean $\mu$ and standard deviation $\sigma$ of the distribution, and $ϵ\sim \mathcal{N}(0,1)$ is sampled to obtain the latent vector as $z_1=\mu +ϵ\sigma$.
  – Diversity Loss:Diversity loss penalizes $\sigma$ for being too far off from zero for unambiguous views, while giving it the liberty to explore the latent space for ambiguous views. $${\mathcal{L}}_{div}=(\sigma -\eta e^{-\frac{({\varphi }_i-{\varphi }_0{)}^2}{{\delta }^2}}{)}^2$$
  其中 ${\varphi }_i$ 是输入图像$I$的方位角，${\varphi }_0$是最大遮挡试图的方位角。
  – For ($c$), $\mathcal{L}={\mathcal{L}}_{lm}+\lambda {\mathcal{L}}_{div}$
  – Inference阶段，改变$ϵ$的取值，可以生成不同的3D点云预测结果。
  
  – 由上图可以看出，如果输入是一张不明确的2D image，调整$ϵ$大小可以得到不同的预测点云结果（如前两行）。对于明确的2D image输入，大小不同的$ϵ$对点云的预测结果影响不大。

3 Result

Ⅱ PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows²

Task

Generate 3D point clouds

Method

Model 3D point clouds as a distribution of distributions. 学习分布的两层层次结构，其中第一层是形状的分布；第二层是给定形状，点的分布。

• Training阶段： 3D点云先经过encoder $Q_{\varphi }$来得到shape representation $z=\mu +\sigma ϵ$. 因为$F_{\varphi }$和$G_{\theta }$都是continuous normalizing flows（是可逆的），所以在训练阶段可以计算形状表征的先验$P_{\varphi }(z)$和$P_{\theta }(X|z)$，从而得到loss ${\mathcal{L}}_{prior}$和${\mathcal{L}}_{recon}$.
• Test阶段：选$\hat{\omega }\sim \mathcal{N}(0,I)$, 然后通过$F_{\varphi }$得到形状表征$\hat{z}=F_{\varphi }(\tilde{\omega })$.
  重复$\tilde{M}$次：
  从 $\mathcal{N}(0,I)$ 分布中取样一个点$\tilde{y}\in {\mathcal{R}}^3$，然后把$\tilde{y}$输入到$G_{\theta }$，得到在形状z上的点$\tilde{x}=G_{\theta }(\tilde{\omega };z)$.

Ⅲ End-to-End Differentiable Learning of Protein structure³

1 Task

Prediction of protein structure from sequence.

2 Highlights

• 用神经网络从氨基酸序列来预测蛋白质结构，不需要co-evolution information.
• 学到了蛋白质序列空间的一种低维度表征

3 Method

（1）用循环神经网络encode蛋白质序列
（2）通过扭转角度参数化局部的蛋白质结构，来使模型推理出不同的构造
（3）通过循环geometric units，把局部的蛋白质结构连接到它的全局表征上
（4）用可导损失函数来刻画预测的结构和真实结构的偏差

Three-stages recurrent geometric network (RGN)

1) computation

输入：每个计算单元一次接收一个residue。（position-specific scoring matrices(PSSMs)?）
输出：每个计算单元输出三个数字，代表输入residue对应的扭转角度（三维）。
computation units are based on LSTM.

2) geometry

输入：扭转角度+由上游geometric单元部分组装完成的骨架
输出：增加了这个residue的骨架

3) assessment

计算预测出的结构和实验结构的差异
Metric: Distance-based root mean square deviation (dRMSD)

---

1. Mandikal P, Murthy N, Agarwal M, et al. 3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image[J]. arXiv preprint arXiv:1807.07796, 2018. ↩︎

2. Yang G, Huang X, Hao Z, et al. PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows[J]. arXiv preprint arXiv:1906.12320, 2019. ↩︎

3. AlQuraishi M. End-to-end differentiable learning of protein structure[J]. Cell systems, 2019, 8(4): 292-301. e3. ↩︎