Introduction to Multimodal Generative Models-Model Architecture Key Features and Codes
1. 多模态生成式模型 MultiModal Generative Models 2024最新列表
Model | Year | Developer | Modality | Architecture | Key Features |
SORA | 2024 | OpenAI | Video,Text | Image Encoder: Diffusion DiT | Generative Modeling,Text-to-Video |
Gemini V1.5 | 2024 | Video,Text,Audio | Image Encoder: ViT,Text Encoder:Transformer | Generative Modeling,Long Context Window | |
BLIP2 | 2023 | Salesforce Research | Image,Text | Q-Former: Bridging Modality Gap,Image Encoder: ViT-L/ViT-G,Text LLM Encoder: OPT/FlanT5 | Generative Modeling,Image-to-Text,Visual Question Answering,Image-to-Text Retrieval |
GPT-4V | 2023 | OpenAI | Image,Text | Text Encoder: GPT | Generative Modeling,Multimodal LLM,Visual Question Answering |
LLaVA | 2023 | Microsoft | Image,Text | Text LLM Encoder: Vicuna,Image Encoder:CLIP visual ViT-L | Generative Modeling,Visual Instruction Generation |
KOSMOS-2 | 2023 | Microsoft | Image,Text | Vision encoder , LLM Encoder: 24-layer MAGNETO Transformer | Multimodal Grounding,Language Understanding and Generation |
PaLM-E | 2023 | Image,Text | Image Encoder: ViT encoding | Multimodal Language Model | |
BLIP | 2022 | Salesforce Research | Image,Text | Image Encoder: ViT-B,ViT-L; Text Encoder: BERT-Base | Generative Modeling,Bootstrapping,VQA,Caption Generation |
FLAMINGO | 2022 | DeepMind | Image,Text | Gated Cross Attention,Multiway Transformer,ViT-giant | VQA,Interleaved Visual and Textual Data |
upCLIP | 2022 | OpenAI | Image,Text | CLIP ViT-L,Diffusion Prior/Autoregressive prior | Generative Modeling,Text-to-Image,Image Generation,Diffusion Models |
BEiT-3 | 2022 | Microsoft | Image,Text | Text Encoder: OPT/FlanT5,Image Encoder:ViT-L/ViT-g | Object Detection,Visual Question Answering,Image Captaining |
CLIP | 2021 | OpenAI | Image,Text | Text Encoder: Transformer; Image Encoder: ResNet/ViT | Multimodal Alignment,Zero-Shot Learning |
ALIGN | 2021 | Image,Text | Image Encoder: EfficientNet,Text-Encoder: BERT | Multimodal Alignment,Image-Text Retrieval |
2.多模态生成式模型常见任务
- Image Captioning
- Image Text Retrieval
- Text-to-Image
- Text-to-Video
- Visual Question Answering
3.相关链接
- SORA: Video generation models as world simulators
- Gemini V1.5: https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf
- BLIP2: BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- GPT-4V: GPT-4 Technical Report
- LLaVa: Visual Instruction Tuning
- KOSMOS-2: KOSMOS-2: Grounding Multimodal Large Language Models to the World
- PaLM-E:PaLM-E: An Embodied Multimodal Language Model
- FLAMINGO:Flamingo: a Visual Language Model for Few-Shot Learning
- UnCLIP:Hierarchical Text-Conditional Image Generation with CLIP Latents
- BEiT3: Image as a Foreign Language: BEIT Pretraining for All Vision and Vision-Language Tasks
- CLIP: Learning Transferable Visual Models From Natural Language Supervision