使用GPT-4V和LlamaIndex构建多模态检索系统

最新推荐文章于 2024-07-27 23:02:02 发布

ppoojjj

最新推荐文章于 2024-07-27 23:02:02 发布

阅读量397

点赞数 3

文章标签： python

本文链接：https://blog.csdn.net/ppoojjj/article/details/140284030

版权

在本文中，我们将展示如何使用LlamaIndex与GPT-4V和CLIP构建一个多模态检索系统。多模态检索系统可以同时处理文本和图像，从而提供更丰富和有用的信息。我们将介绍以下步骤：

使用多模态LLM GPT4V类处理多张图片
从相关的维基百科文章和SEC 10K报告下载文本、图像和PDF原始文件
为文本和图像构建多模态索引和向量存储
根据第一步的图像推理同时检索相关文本和图像

安装依赖包

首先，我们需要安装一些必要的Python包：

%pip install llama-index-multi-modal-llms-openai
%pip install llama-index-vector-stores-qdrant

%pip install llama_index ftfy regex tqdm
%pip install git+https://github.com/openai/CLIP.git
%pip install torch torchvision
%pip install matplotlib scikit-image
%pip install -U qdrant_client

下载图片

我们将从Tesla网站下载一些图片，作为示例数据。

from pathlib import Path

input_image_path = Path("input_images")
if not input_image_path.exists():
    Path.mkdir(input_image_path)

!wget "https://docs.google.com/uc?export=download&id=1nUhsBRiSWxcVQv8t8Cvvro8HJZ88LCzj" -O ./input_images/long_range_spec.png
!wget "https://docs.google.com/uc?export=download&id=19pLwx0nVqsop7lo0ubUSYTzQfMtKJJtJ" -O ./input_images/model_y.png
!wget "https://docs.google.com/uc?export=download&id=1utu3iD9XEgR5Sb7PrbtMf1qw8T1WdNmF" -O ./input_images/performance_spec.png
!wget "https://docs.google.com/uc?export=download&id=1dpUakWMqaXR4Jjn1kHuZfB0pAXvjn2-i" -O ./input_images/price.png
!wget "https://docs.google.com/uc?export=download&id=1qNeT201QAesnAP5va1ty0Ky5Q_jKkguV" -O ./input_images/real_wheel_spec.png

使用GPT4V理解这些输入图片

from llama_index.multi_modal_llms.openai import OpenAIMultiModal
from llama_index.core import SimpleDirectoryReader

# 设置API Token
OPENAI_API_TOKEN = ""
os.environ["OPENAI_API_KEY"] = OPENAI_API_TOKEN

# 加载图像数据
image_documents = SimpleDirectoryReader("./input_images").load_data()

# 初始化GPT-4V模型
openai_mm_llm = OpenAIMultiModal(
    model="gpt-4-vision-preview", api_key=OPENAI_API_TOKEN, max_new_tokens=1500
)

# 生成描述
response_1 = openai_mm_llm.complete(
    prompt="Describe the images as an alternative text",
    image_documents=image_documents,
)

print(response_1)

构建多模态索引和向量存储

from llama_index.core.indices import MultiModalVectorStoreIndex
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.core import SimpleDirectoryReader, StorageContext

import qdrant_client


# 创建本地Qdrant向量存储
client = qdrant_client.QdrantClient(path="qdrant_mm_db")

text_store = QdrantVectorStore(
    client=client, collection_name="text_collection"
)
image_store = QdrantVectorStore(
    client=client, collection_name="image_collection"
)
storage_context = StorageContext.from_defaults(
    vector_store=text_store, image_store=image_store
)

# 创建多模态索引
documents = SimpleDirectoryReader("./mixed_wiki/").load_data()
index = MultiModalVectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
)

检索和查询多模态索引中的文本和图像

# 生成文本检索结果
MAX_TOKENS = 50
retriever_engine = index.as_retriever(
    similarity_top_k=3, image_similarity_top_k=3
)

retrieval_results = retriever_engine.retrieve(response_2.text[:MAX_TOKENS])

from llama_index.core.response.notebook_utils import display_source_node
from llama_index.core.schema import ImageNode

retrieved_image = []
for res_node in retrieval_results:
    if isinstance(res_node.node, ImageNode):
        retrieved_image.append(res_node.node.metadata["file_path"])
    else:
        display_source_node(res_node, source_length=200)

plot_images(retrieved_image)