[全面解析：如何使用Facebook Messenger数据进行AI模型微调]-CSDN博客

本文链接：https://blog.csdn.net/tt_jishu/article/details/142428748

# 全面解析：如何使用Facebook Messenger数据进行AI模型微调

## 引言

在AI模型训练中，个性化数据具有重要意义。本文将详细介绍如何从Facebook Messenger导入数据，并利用这些数据对AI模型进行微调。无论你是初学者还是经验丰富的开发者，这篇文章都将为你提供实用的指导。

## 主要内容

### 1. 下载Messenger数据

要开始，我们需要将Messenger数据下载为JSON格式。请参考[官方指南](https://www.facebook.com/help)进行下载。为了演示，我们还提供了一个示例数据集可以在[此Google Drive链接](https://drive.google.com/file/d/1rh1s1o2i7B-Sk1v9o8KNgivLVGwJ-osV/view?usp=sharing)找到。

下载和解压缩数据的代码如下：

```python
import zipfile
import requests

def download_and_unzip(url: str, output_path: str = "file.zip") -> None:
    file_id = url.split("/")[-2]
    download_url = f"https://drive.google.com/uc?export=download&id={file_id}"

    response = requests.get(download_url)
    if response.status_code != 200:
        print("Failed to download the file.")
        return

    with open(output_path, "wb") as file:
        file.write(response.content)
        print(f"File {output_path} downloaded.")

    with zipfile.ZipFile(output_path, "r") as zip_ref:
        zip_ref.extractall()
        print(f"File {output_path} has been unzipped.")

url = "https://drive.google.com/file/d/1rh1s1o2i7B-Sk1v9o8KNgivLVGwJ-osV/view?usp=sharing"
download_and_unzip(url)

2. 创建Chat Loader

我们可以使用FolderFacebookMessengerChatLoader类加载整个目录中的聊天记录，或使用SingleFileFacebookMessengerChatLoader类加载单个文件。

from langchain_community.chat_loaders.facebook_messenger import (
    FolderFacebookMessengerChatLoader,
    SingleFileFacebookMessengerChatLoader,
)

loader = SingleFileFacebookMessengerChatLoader(
    path="./hogwarts/inbox/HermioneGranger/messages_Hermione_Granger.json",
)

chat_session = loader.load()[0]
print(chat_session["messages"][:3])

3. 准备微调数据

我们可以合并同一发送者的消息，然后选择某个发送者的消息作为AI。接下来使用这些数据进行模型微调。

from langchain_community.chat_loaders.utils import (
    map_ai_messages,
    merge_chat_runs,
)

merged_sessions = merge_chat_runs(chat_sessions)
alternating_sessions = list(map_ai_messages(merged_sessions, "Harry Potter"))

from langchain_community.adapters.openai import convert_messages_for_finetuning

training_data = convert_messages_for_finetuning(alternating_sessions)
print(f"Prepared {len(training_data)} dialogues for training")

4. 微调模型

确保已安装openai并设置OPENAI_API_KEY。

%pip install --upgrade --quiet langchain-openai

import json
import time
from io import BytesIO
import openai

my_file = BytesIO()
for m in training_examples:
    my_file.write((json.dumps({"messages": m}) + "\n").encode("utf-8"))

my_file.seek(0)
training_file = openai.files.create(file=my_file, purpose="fine-tune")

status = openai.files.retrieve(training_file.id).status
start_time = time.time()
while status != "processed":
    time.sleep(5)
    status = openai.files.retrieve(training_file.id).status

job = openai.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-3.5-turbo",
)

5. 在LangChain中使用

使用微调后的模型ID直接在ChatOpenAI模型类中调用。

from langchain_openai import ChatOpenAI

model = ChatOpenAI(
    model=job.fine_tuned_model,
    temperature=1,
)

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([("human", "{input}")])
chain = prompt | model | StrOutputParser()

for tok in chain.stream({"input": "What classes are you taking?"}):
    print(tok, end="")