Contrastive Representation Learning

Contrastive Representation Learning

https://lilianweng.github.io/lil-log/2021/05/31/contrastive-representation-learning.html 

The main idea of contrastive learning is to learn representations such that similar samples stay close to each other, while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised data and has been shown to achieve good performance on a variety of vision and language tasks.

The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. Contrastive learning can be applied to both supervised and unsupervised settings. When working with unsupervised data, contrastive learning is one of the most powerful approaches in self-supervised learning.

• Contrastive Training Objectives
  • Contrastive Loss
  • Triplet Loss
  • Lifted Structured Loss
  • N-pair Loss
  • NCE
  • InfoNCE
  • Soft-Nearest Neighbors Loss
  • Common Setup
• Key Ingredients
  • Heavy Data Augmentation
  • Large Batch Size
  • Hard Negative Mining
• Vision: Image Embedding
  • Image Augmentations
  • Parallel Augmentation
  • Memory Bank
  • Feature Clustering
  • Working with Supervised Datasets
• Language: Sentence Embedding
  • Text Augmentation
  • Supervision from NLI
  • Unsupervised Sentence Embedding Learning
• References