对比预测编码用于表征学习

论文题目：Representation Learning with Contrastive Predictive Coding

Summary

• 论文通过设计一个编码器和一个自回归模型，使的模型能够提取上下文信息，对当前特征进行编码，同时能利用当前表征及以前的表征来预测未来的表征。

Problem Statement

• 监督模型由于有庞大的训练集进行支撑，因此可以有较好的效果，但无监督学习认存在较大的挑战，如何学习得到通用的表征模型，是当前需要解决的问题。

Method

• 模型：编码器+自回归模型
  
• 利用噪声对比估计损失函数进行优化网络
  

Evaluation

• 为评估模型的好坏以及通用性，评估方法如下：
• 音频
• 视觉
• 自然语言
• 辅助强化学习

Conclusion

• 将编码器和自回归模型相结合使的模型可以学习得到当前的表征和上下文信息，因此可用于多种数据类型的处理。

Notes

• While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence.
• Improving representation learning requires features that are less specialized towards solving a
  single supervised task.
• Despite its importance, unsupervised learning is yet to see a breakthrough similar to supervised
  learning: modeling high-level representations from raw observations remains elusive.

References

• Unsupervised visual representation learning by context prediction.
• Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.
• Adaptive importance sampling to accelerate training of a neural probabilistic language model.
• Distance metric learning for large margin nearest neighbor classification.
• Visualizing data using t-sne
• Multi-task self-supervised visual learning.
• Layer normalization.