Self-supervised Learning: Generative or Contrastive总结

Self-supervised Learning: Generative or Contrastive
type	method	fos	model	self-supervision	pretext-task	特点	备注
generative self-supervised learning	Auto-regressive (AR) Model	nlp	GPT/GPT-2	follow words	next-word prediction	1. 采用逐项生成 (component-by-component) 的方式来生成完整的数据。 2. advantage:model the context dependency well 3. disadvantages:access its context from one direction
		cv	PixelRNN/PixelCNN(image)/WavNet(video)	follow pixels	next-pixel prediction
		graph	GraphRNN/MRNN/GCPN(reinforcement learning)	follow edges	next-edge prediction
	Flow-based Model	cv	NICE/RealNVP/Glow	whole image	Image reconstruction	从数据中估计复杂的高维密度函数。通过一系列可逆变换来进行生成。由于这些变换都是可逆的，因此隐变量（输入）和生成结果（输出）之间可以任意转换
	Auto-encoding (AE) Model	NLP	FastText/word2vec	Context words	CBOW & SkipGram/CBOW	Context Prediction Model (CPM):基于输入预测上下文信息
		graph	DeepWalk：基于随机游走	Graph edges	Graph edges
			LINE：基于邻居汇聚
		NLP	BERT/ SpanBert/Ernie/MLM	Masked words ,Sentence topic/ Masked words	Masked language model,Next senetence prediction /Masked language model	Denoising AE Model：representation should be robust to the introduction of noise.
		Graph	GPT-GNN	Attribute & Edge	Masked graph generation
		CV	VAE/VQ-VAE/VQ-VAE2	Whole image	Image reconstruction	Variational AE Model(变分自编码器）：通过对输入信息进行重构，来学习输入信息的表征的均值和方差
		Graph	VGAE/DVNE/vGraph	Graph edges	Graph edges
	Hybrid Generative Models	nlp	XLNet	Masked words	Permutation language model	AE+AR
		graph	Graph	Attribute & Edge	Sequential graph generation	Flow+AR
		cv	MADE			AE+AR
constractive self-supervised learning:通过对比正负样本来学习表示	Context-Instance Contrast:上下文与实例的对比，也就是建立局部与全局之间的关联	cv	RotNet/PIRL	Spatial relations	Rotation Prediction/Jigsaw	predict relative position(PRP):许多数据在其各部分之间包含丰富的空间或序列关系。	区别： 1）PRP 关注于学习局部组件之间的相对位置。全局上下文是预测这些关系的隐含要求(例如，了解大象的长相对于预测它头和尾巴的相对位置至关重要)； 2）MI 侧重于学习局部组件和全局上下文之间的明确归属关系。局部之间的相对位置被忽略。
		nlp	ALBERT	Masked words,Sentence order	Masked language model, Sentence order prediction
		cv	Deep InfoMax/CPC/AMDIM	Belonging	MI Maximization	Maximize Nutual Information(MI):focus on learning the explicit belonging relationship betweenlocal parts and global context.
		nlp	InfoWorld
		graph	InfoGraph/DeelGraph InfoMax
	Instance-Instace Constract:实例级表示，而不是上下文级表示，对于广泛的分类任务来说更为关键	cv	DeepCluster/Local Aggregation/ClusterFit	Similarity (Instance-Instance)	Cluster discrimination(基于聚类判别)	首先研究基于聚类的实例与实例之间的对比，也就是利用聚类产生伪标签，这种方法不行。
			MoCo/SimCLR/InfoMin/BYOL/SimSiam	Identity (Instance-Instance)	Instance discrimination(基于实例判别)	moco打破了一正一负，大大增加了负样本的数量。
		graph	GCC/CMC-Graph/M3S
Generative-Contrastive (Adversarial)	Generate  with Complete Input	cv	VAE-GAN/BI-GAN/AAE/ALI	whole image	Image reconstruction	GAN和GAN的变体，目的就是捕获完整信息
	Recover with Partial Input	CV	Colorization	Image color	Colorization	Recover with Partial Input
			Inpainting	Parts of images	Inpainting
			Super-Resolution	Details of images	Super-Resolution
	Pretrained Langrage Model	nlp	ELECTRA/WKLM			Pretrained Langrage Model
	Graph learning	Graph	ANE/GraphGAN/GraphSCAN			Graph learning