多模态生成式模型MultiModal Generative Models 2024最新列表

Introduction to Multimodal Generative Models-Model Architecture Key Features and Codes

1. 多模态生成式模型 MultiModal Generative Models 2024最新列表

Model	Year	Developer	Modality	Architecture	Key Features
SORA	2024	OpenAI	Video,Text	Image Encoder: Diffusion DiT	Generative Modeling,Text-to-Video
Gemini V1.5	2024	Google	Video,Text,Audio	Image Encoder: ViT,Text Encoder:Transformer	Generative Modeling,Long Context Window
BLIP2	2023	Salesforce Research	Image,Text	Q-Former: Bridging Modality Gap,Image Encoder: ViT-L/ViT-G,Text LLM Encoder: OPT/FlanT5	Generative Modeling,Image-to-Text,Visual Question Answering,Image-to-Text Retrieval
GPT-4V	2023	OpenAI	Image,Text	Text Encoder: GPT	Generative Modeling,Multimodal LLM,Visual Question Answering
LLaVA	2023	Microsoft	Image,Text	Text LLM Encoder: Vicuna,Image Encoder:CLIP visual ViT-L	Generative Modeling,Visual Instruction Generation
KOSMOS-2	2023	Microsoft	Image,Text	Vision encoder , LLM Encoder: 24-layer MAGNETO Transformer	Multimodal Grounding,Language Understanding and Generation
PaLM-E	2023	Google	Image,Text	Image Encoder: ViT encoding	Multimodal Language Model
BLIP	2022	Salesforce Research	Image,Text	Image Encoder: ViT-B,ViT-L; Text Encoder: BERT-Base	Generative Modeling,Bootstrapping,VQA,Caption Generation
FLAMINGO	2022	DeepMind	Image,Text	Gated Cross Attention,Multiway Transformer,ViT-giant	VQA,Interleaved Visual and Textual Data
upCLIP	2022	OpenAI	Image,Text	CLIP ViT-L,Diffusion Prior/Autoregressive prior	Generative Modeling,Text-to-Image,Image Generation,Diffusion Models
BEiT-3	2022	Microsoft	Image,Text	Text Encoder: OPT/FlanT5,Image Encoder:ViT-L/ViT-g	Object Detection,Visual Question Answering,Image Captaining
CLIP	2021	OpenAI	Image,Text	Text Encoder: Transformer; Image Encoder: ResNet/ViT	Multimodal Alignment,Zero-Shot Learning
ALIGN	2021	Google	Image,Text	Image Encoder: EfficientNet,Text-Encoder: BERT	Multimodal Alignment,Image-Text Retrieval

2.多模态生成式模型常见任务

• Image Captioning
• Image Text Retrieval
• Text-to-Image
• Text-to-Video
• Visual Question Answering

3.相关链接

• SORA: Video generation models as world simulators
• Gemini V1.5: https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf
• BLIP2: BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• GPT-4V: GPT-4 Technical Report
• LLaVa: Visual Instruction Tuning
• KOSMOS-2: KOSMOS-2: Grounding Multimodal Large Language Models to the World
• PaLM-E：PaLM-E: An Embodied Multimodal Language Model
• FLAMINGO：Flamingo: a Visual Language Model for Few-Shot Learning
• UnCLIP：Hierarchical Text-Conditional Image Generation with CLIP Latents
• BEiT3: Image as a Foreign Language: BEIT Pretraining for All Vision and Vision-Language Tasks
• CLIP: Learning Transferable Visual Models From Natural Language Supervision

参考文档：多模态生成式模型MultiModal Generative Models 2024最新列表 - 知乎