使用iMessage Chat Loader进行对话数据转换与模型微调

最新推荐文章于 2025-06-13 15:03:33 发布

qahaj

最新推荐文章于 2025-06-13 15:03:33 发布

阅读量343

点赞数 4

文章标签： oracle 数据库 python

本文链接：https://blog.csdn.net/qahaj/article/details/145767212

版权

在本文中，我们将探讨如何使用iMessage Chat Loader来将iMessage聊天记录转换为LangChain可用的聊天消息格式，并进一步用于模型微调。具体来说，我们将查看如何从macOS系统的聊天数据库中加载数据，并利用这些数据对模型进行微调。

技术背景介绍

在macOS上，iMessage的聊天记录被保存在~/Library/Messages/chat.db这个SQLite数据库文件中。我们可以通过访问这个数据库来提取聊天记录，用于机器学习和自然语言处理任务。本文示例将展示如何使用IMessageChatLoader从该数据库中加载数据并进行处理。

核心原理解析

IMessageChatLoader是一个工具类，用于从chat.db文件中读取iMessage对话内容，并将其转换为LangChain的消息对象格式。可以通过以下步骤实现聊天记录的加载和转换：

访问Chat DB：由于安全设置，直接访问~/Library/Messages路径可能会遇到权限问题，因此建议将数据库复制到一个可访问的目录。
加载器创建：将数据库路径提供给IMessageChatLoader，并创建加载器实例。
消息加载和处理：调用load()或lazy_load()方法加载消息，可以选择合并连续的消息或将特定发送者的消息转换为AI消息。

代码实现演示

1. 下载示例数据库

import requests

def download_drive_file(url: str, output_path: str = "chat.db") -> None:
    file_id = url.split("/")[-2]
    download_url = f"https://drive.google.com/uc?export=download&id={file_id}"

    response = requests.get(download_url)
    if response.status_code != 200:
        print("Failed to download the file.")
        return

    with open(output_path, "wb") as file:
        file.write(response.content)
        print(f"File {output_path} downloaded.")

url = "https://drive.google.com/file/d/1NebNKqTA2NXApCmeH6mu0unJD2tANZzo/view?usp=sharing"
download_drive_file(url)

2. 创建聊天加载器

from langchain_community.chat_loaders.imessage import IMessageChatLoader

loader = IMessageChatLoader(
    path="./chat.db",
)

3. 加载和处理消息

from typing import List
from langchain_community.chat_loaders.utils import map_ai_messages, merge_chat_runs
from langchain_core.chat_sessions import ChatSession

raw_messages = loader.lazy_load()
merged_messages = merge_chat_runs(raw_messages)

chat_sessions: List[ChatSession] = list(
    map_ai_messages(merged_messages, sender="Tortoise")
)

print(chat_sessions[0]["messages"][:3])

4. 准备微调数据

from langchain_community.adapters.openai import convert_messages_for_finetuning

training_data = convert_messages_for_finetuning(chat_sessions)
print(f"Prepared {len(training_data)} dialogues for training")

5. 微调模型

import json
import time
from io import BytesIO
import openai

# We will write the jsonl file in memory
my_file = BytesIO()
for m in training_data:
    my_file.write((json.dumps({"messages": m}) + "\n").encode("utf-8"))

my_file.seek(0)
training_file = openai.files.create(file=my_file, purpose="fine-tune")

# OpenAI audits each training file for compliance reasons.
status = openai.files.retrieve(training_file.id).status
start_time = time.time()
while status != "processed":
    print(f"Status=[{status}]... {time.time() - start_time:.2f}s", end="\r", flush=True)
    time.sleep(5)
    status = openai.files.retrieve(training_file.id).status
print(f"File {training_file.id} ready after {time.time() - start_time:.2f} seconds.")

job = openai.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-3.5-turbo",
)

status = openai.fine_tuning.jobs.retrieve(job.id).status
start_time = time.time()
while status != "succeeded":
    print(f"Status=[{status}]... {time.time() - start_time:.2f}s", end="\r", flush=True)
    time.sleep(5)
    job = openai.fine_tuning.jobs.retrieve(job.id)
    status = job.status

print(job.fine_tuned_model)

6. 在LangChain中使用

from langchain_openai import ChatOpenAI

model = ChatOpenAI(
    model=job.fine_tuned_model,
    temperature=1,
)

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are speaking to hare."),
        ("human", "{input}"),
    ]
)

chain = prompt | model | StrOutputParser()

for tok in chain.stream({"input": "What's the golden thread?"}):
    print(tok, end="", flush=True)