连理o的博客

[ICML 2021] CLIP: Learning Transferable Visual Models From Natural Language Supervision

各种借鉴了 CLIP 的模型

字节提出的多模态模型 x-vlm

Seeing out of the box: End-to-end pre-training for vision-language representation learning

ContentsIntroductionApproachModel ArchitecturePretext TasksExperimentsPretraining CorpusDownstream TasksAblation Study & Visualization AnalysisReferencesIntroduction自监督视觉语言预训练 (vision-and-language pretraining, VLP) 旨在从大规模图像-文本数据中学习可迁移的多模态特征。主流的 VLP

目录IntroductionModel ArchitecturePre-training tasksPre-training datasetsExperimentsReferencesIntroductionUNITER: a UNiversal Image-TExt Representation, which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.Model Architectu

VL-BERT 其实就是一个增加了视觉元素输入的多模态 BERT，它能高效地融合对齐视觉和语言信息，可以被当作大多数视觉-语言下游任务的预训练模型

目录IntroductionReferencesIntroductionWe present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning.VILLA consists of two training stages: (i) task-agnostic adversarial pre-training; fo

目录IntroductionScene Graph (场景图)ERNIE-ViLModel ArchitectureScene Graph Prediction (SGP)ExperimentsTraining ERNIE-ViLDownstream TasksResultsReferencesIntroductionScene Graph (场景图)Scene graphs contain structured knowledge of visual scenes, including the p

目录Model ArchitecturePre-Training StrategiesExperimental Setup and ResultsReferencesLXMERT: Learning Cross-Modality Encoder Represen-tations from TransformersModel ArchitectureInput Embeddings: input embedding layers 负责将 sentence 和 image 分别转化为 word-l

目录ViLBERT: Extending BERT to Jointly Represent Images and TextExperimental SettingsReferencesViLBERT: Vision-and-Language BERTViLBERT: Extending BERT to Jointly Represent Images and TextTwo-stream Architecture: ViLBERT 采用 two-stream 架构，由两个并行的 BERT-st

目录VisualBERTExperimentReferencesVisualBERTArchitecture网络架构与 BERT 相同。文本部分的处理也与 BERT 相同，下面主要介绍视觉图像部分的处理：设 FFF 为 visual embeddings 的集合，f∈Ff\in Ff∈F 为一个 bounding region 对应的 feature，为如下 3 部分 embedding 的加和：(1) fof_ofo​: bounding region 对于的 visual feature re

目录Oscar Pre-trainingAdapting to V+L TasksExperimental Results & AnalysisPerformance Comparison with SoTAQualitative StudiesReferencesOSCAR: Object-SemantiCs Aligned pRe-trainingOscar Pre-trainingInputOscar 将每个输入的 image-text pair 都表示为 Word-Tag-I