实现图像描述生成与索引查询：从入门到实践

aehrutktrjk

于 2024-10-03 10:10:07 发布

阅读量177

点赞数 1

文章标签： python 开发语言

本文链接：https://blog.csdn.net/aehrutktrjk/article/details/142689453

版权

实现图像描述生成与索引查询：从入门到实践

引言

在人工智能领域，图像描述生成一直是一个热门话题。通过自动生成图像描述，我们可以更高效地管理和检索图像数据。本篇文章将介绍如何使用预训练的Salesforce BLIP图像描述模型，结合Langchain框架，实现图像描述生成与查询的完整流程。

主要内容

1. 准备工作

首先，我们需要安装必要的Python包：

%pip install -qU transformers langchain_openai langchain_chroma

然后，设置OpenAI API Key：

import getpass
import os

os.environ["OPENAI_API_KEY"] = getpass.getpass()  # 输入您的OpenAI API密钥

2. 准备图像数据

我们从Wikimedia准备一组图像URL以进行处理：

list_image_urls = [
    "https://upload.wikimedia.org/wikipedia/commons/thumb/e/ec/Ara_ararauna_Luc_Viatour.jpg/1554px-Ara_ararauna_Luc_Viatour.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/thumb/0/0c/1928_Model_A_Ford.jpg/640px-1928_Model_A_Ford.jpg",
]

3. 使用ImageCaptionLoader生成图像描述

from langchain_community.document_loaders import ImageCaptionLoader

loader = ImageCaptionLoader(images=list_image_urls)
list_docs = loader.load()

print(list_docs)

此步骤将每个图像转换为一个带有描述的文档对象。

4. 构建索引并进行查询

首先，使用Chroma构建向量存储：

from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
splits = text_splitter.split_documents(list_docs)
vectorstore = Chroma.from_documents(documents=splits, embedding=OpenAIEmbeddings())
retriever = vectorstore.as_retriever(k=2)

然后，我们可以创建检索链并进行查询：

from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4o", temperature=0)

system_prompt = (
    "You are an assistant for question-answering tasks. "
    "Use the following pieces of retrieved context to answer "
    "the question. If you don't know the answer, say that you "
    "don't know. Use three sentences maximum and keep the "
    "answer concise."
    "\n\n"
    "{context}"
)

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system_prompt),
        ("human", "{input}"),
    ]
)

question_answer_chain = create_stuff_documents_chain(model, prompt)
rag_chain = create_retrieval_chain(retriever, question_answer_chain)

response = rag_chain.invoke({"input": "What animals are in the images?"})
print(response["answer"])