ECCV 2020 workshop self-supervision note

Self Supervised Learning: What is Next?

一、Perspectives on Unsupervised Representation Learning

1. SimCLR seems to benefit from deep models and long training more than SL
2. how we handle the weight initialization may play a big role in the final features
3. We also should probably move away from comparing to supervised learning features as they may not be the golden standard (e.g., mid-range attributes)
4. Probably a combination of both discriminative and aligning principles (through multi task learning) is a plausible direction (see also Feng et al, 2019)

二、Beyond Self-Supervised Representation Learning

1. Three phases of self-supervised learning: Classical – Expansion – Uncurated; Uncurated is less explored.
2. Self-supervision from uncurated data, i.e. no pre-defined datasets, instead: – Random YouTube videos, so not class balanced, long tailed – Daily life videos, e.g. Vlogs, babycams
3. New learning schedules: • Curriculum learning • How to obtain informative (hard) samples
4. Universal networks: able to ingest multiple-modalities and carry out multiple tasks
5. Curated datasets still have their uses: become new evaluation benchmarks

三、Self-Supervision as a Path to a Post-Dataset Era

1. Data is infinite. Test set is contemporary. So network always can find a way to treat test set. We need to get rid of it.
2. Real-world motivation: Biological agents never see the same data twice! Repeating the same sample might encourage memorization/discourage generalization.
3. Self-supervision can help adapt to new domain.
4. Inspiration: one sample learning (An image specific )
5. Datasets were useful but they might be hunting us and thus missing something important.
6. Let's start thinking beyond fixed train/test datasets, and more about continuous data streams (which happen naturally and do not need some designs such as crop or data augmentation)----videos