更好的理解CLIP模型

大路诗人路小果

已于 2024-03-26 10:20:49 修改

阅读量588

点赞数 4

文章标签： transformer

于 2024-03-25 14:47:59 首次发布

本文链接：https://blog.csdn.net/a14285700/article/details/137005265

版权

CLIP模型使用(ViT+BERT)

这个ViT+BERT只说明最简单的CLIP文本输出格式，它的输出有什么，之后的文章将会引出两个模态的 embeds 从何而来，它是如何从模型的输出特征中提取出来，为此模型实际上又做了什么改变

在这里插入图片描述

from transformers import CLIPProcessor, CLIPModel
from PIL import Image
from IPython import embed
from transformers import CLIPTokenizer
import torch

model_name = "/public_bme/data/breast-10-12/CausalFromText/clip-vit-b-16/"
model = CLIPModel.from_pretrained(model_name)
tokenizer = CLIPTokenizer.from_pretrained(model_name)
processor = CLIPProcessor.from_pretrained(model_name)

texts = ["This is a description of the first image.", "New as a word of the good fuck."]
images = [Image.open("CLIP.png"), Image.open("CLIP.png")]
inputs_text = tokenizer(texts, return_tensors="pt", padding=True, truncation=True, max_length=512