NVIDIA AI-AGENT夏季训练营
项目名称:AI-AGENT夏季训练营 — 基于小模型(SLM)-Phi3实现图片文件转描述信息的前端页面实现
报告日期:2024年8月18日
项目负责人:Yutong Bao
项目概述:
应用场景: 本项目展示了基于英伟达的基于 NIM 建构多模态 AI-Agent的创新应用。从一个图片当中,识别其关键信息,输出这张照片的内容及含义,部署至前端页面。
技术方案与实施步骤
模型选择:本项目采用Microsoft的【phi-3-vision-128k-instruct】来解析图片数据,未来可以考虑项目可以落地到jetson系列的设备进行边缘计算。
数据的构建: 将jpg、png等格式的图片文件转换为base64格式,方便python的IO读取。
实施步骤:
环境搭建: 需要提前安装python环境并导包
import gradio as gr
import os
import base64
from langchain_nvidia_ai_endpoints import ChatNVIDIA
from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings
from langchain.chains import ConversationalRetrievalChain, LLMChain
from langchain.chains.conversational_retrieval.prompts import CONDENSE_QUESTION_PROMPT, QA_PROMPT
from langchain.chains.question_answering import load_qa_chain
from langchain.memory import ConversationBufferMemory
from langchain.vectorstores import FAISS
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.prompts import ChatPromptTemplate
具体完整可运行代码如下
import gradio as gr
import os
import base64
from langchain_nvidia_ai_endpoints import ChatNVIDIA
from langchain_nvidia_ai_endpoints import NVIDIAEmbeddings
from langchain.chains import ConversationalRetrievalChain, LLMChain
from langchain.chains.conversational_retrieval.prompts import CONDENSE_QUESTION_PROMPT, QA_PROMPT
from langchain.chains.question_answering import load_qa_chain
from langchain.memory import ConversationBufferMemory
from langchain.vectorstores import FAISS
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.prompts import ChatPromptTemplate
os.environ["NVIDIA_API_KEY"] = "your API_KEY"
image_file = "Downloads/2024暑期训练营/2024_summer_bootcamp/day1/sky.jpg"
def image2b64(image_file):
with open(image_file, "rb") as f:
image_b64 = base64.b64encode(f.read()).decode()
return image_b64
def chart_agent_gr(image_b64, user_input, image):
image_b64 = image2b64(image_b64)
# Chart reading Runnable
chart_reading = ChatNVIDIA(model="microsoft/phi-3-vision-128k-instruct")
chart_reading_prompt = ChatPromptTemplate.from_template(
'Generate descriptive text for the following images, : <img src="data:image/png;base64,{image_b64}" />'
)
chart_chain = chart_reading_prompt | chart_reading
return chart_chain.invoke({"image_b64": image_b64, "input": user_input, "image": image})
multi_modal_chart_agent = gr.Interface(fn=chart_agent_gr,
inputs=[gr.Image(label="Upload image", type="filepath"), 'text'],
outputs=['text'],
title="Multi Modal chat agent",
description="Multi Modal chat agent",
allow_flagging="never")
multi_modal_chart_agent.launch(debug=True, share=False, show_api=False, server_port=1234, server_name="0.0.0.0")
项目成果展示
上传照片后,submit,右边显示对应图片的信息内容,后续方便继续操作。
项目总结与展望
后续考虑将其改进为更多功能,更加细致的图片信息抽取及智能应用网页
百尺竿头更进一步!