使用Facebook Messenger数据进行AI模型微调的完整指南

AI程序猿人

于 2024-10-10 12:00:54 发布

阅读量540

点赞数 11

文章标签： facebook 人工智能数据库模型微调 AI 大模型大模型技术

本文链接：https://blog.csdn.net/python1222_/article/details/142818081

版权

引言

在这个数字化时代，个性化的AI助手已经成为热门话题。而微调模型是实现这一目标的核心方法之一。本文将介绍如何从Facebook Messenger中提取对话数据，并使用这些数据对AI模型进行微调，从而创建更符合个人需求的AI助手。

主要内容

1. 下载数据

要开始使用，首先需要下载Facebook Messenger的数据。可以按照此处说明下载数据，确保选择JSON格式。

我们提供了一个示例数据集，可以通过以下代码下载和解压：

import zipfile
import requests

def download_and_unzip(url: str, output_path: str = "file.zip") -> None:
    file_id = url.split("/")[-2]
    download_url = f"https://drive.google.com/uc?export=download&id={file_id}"

    response = requests.get(download_url)
    if response.status_code != 200:
        print("Failed to download the file.")
        return

    with open(output_path, "wb") as file:
        file.write(response.content)
        print(f"File {output_path} downloaded.")

    with zipfile.ZipFile(output_path, "r") as zip_ref:
        zip_ref.extractall()
        print(f"File {output_path} has been unzipped.")

url = "https://drive.google.com/file/d/1rh1s1o2i7B-Sk1v9o8KNgivLVGwJ-osV/view?usp=sharing"
download_and_unzip(url)

2. 创建聊天加载器

我们可以使用FolderFacebookMessengerChatLoader或SingleFileFacebookMessengerChatLoader加载数据：

from langchain_community.chat_loaders.facebook_messenger import (
    FolderFacebookMessengerChatLoader,
    SingleFileFacebookMessengerChatLoader,
)

loader = SingleFileFacebookMessengerChatLoader(
    path="./hogwarts/inbox/HermioneGranger/messages_Hermione_Granger.json",
)

chat_session = loader.load()[0]
print(chat_session["messages"][:3])

3. 准备微调

合并消息并将指定发送者的消息转换为“AIMessage”类：

from langchain_community.chat_loaders.utils import (
    map_ai_messages,
    merge_chat_runs,
)

merged_sessions = merge_chat_runs(chat_sessions)
alternating_sessions = list(map_ai_messages(merged_sessions, "Harry Potter"))

from langchain_community.adapters.openai import convert_messages_for_finetuning
training_data = convert_messages_for_finetuning(alternating_sessions)
print(f"Prepared {len(training_data)} dialogues for training")

4. 微调模型

确保安装了openai库并设置了OPENAI_API_KEY：

import json
import time
from io import BytesIO
import openai

my_file = BytesIO()
for m in training_examples:
    my_file.write((json.dumps({"messages": m}) + "\n").encode("utf-8"))

my_file.seek(0)
training_file = openai.files.create(file=my_file, purpose="fine-tune")

status = openai.files.retrieve(training_file.id).status
start_time = time.time()
while status != "processed":
    print(f"Status=[{status}]... {time.time() - start_time:.2f}s", end="\r", flush=True)
    time.sleep(5)
    status = openai.files.retrieve(training_file.id).status
print(f"File {training_file.id} ready after {time.time() - start_time:.2f} seconds.")

5. 在LangChain中使用

使用生成的模型ID直接在ChatOpenAI类中：

from langchain_openai import ChatOpenAI

model = ChatOpenAI(
    model=job.fine_tuned_model,
    temperature=1,
)

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages(
    [
        ("human", "{input}"),
    ]
)

chain = prompt | model | StrOutputParser()