深入掌握Confident DeepEval：为LLM构建强大的单元测试

最新推荐文章于 2024-10-01 05:26:49 发布

bhawfgrcbtwny

最新推荐文章于 2024-10-01 05:26:49 发布

阅读量378

点赞数 4

文章标签： chrome 前端 python

本文链接：https://blog.csdn.net/bhawfgrcbtwny/article/details/142428516

版权

引言

构建鲁棒的语言模型（LLM）需要不断的测试和迭代。Confident的DeepEval包通过提供从合成数据创建到性能测试的全面支持，实现了这一目标。在本指南中，我们将演示如何使用DeepEval对LLM进行性能测试，以及如何定义自定义指标并将其记录在仪表盘中。

主要内容

安装和设置

首先，我们需要安装所需的库：

%pip install --upgrade --quiet langchain langchain-openai langchain-community deepeval langchain-chroma

获取API凭证

访问 Confident
点击“Organization”
复制API Key
登录时，还需要设置一个implementation name

!deepeval login

设置DeepEval

使用DeepEvalCallbackHandler来配置要跟踪的指标，目前支持答案相关性、偏见和毒性。

from deepeval.metrics.answer_relevancy import AnswerRelevancy

# 确保答案的相关性至少达到0.5
answer_relevancy_metric = AnswerRelevancy(minimum_score=0.5)

使用DeepEval进行测试

场景1：将DeepEval集成到LLM中

from langchain_community.callbacks.confident_callback import DeepEvalCallbackHandler
from langchain_openai import OpenAI

deepeval_callback = DeepEvalCallbackHandler(
    implementation_name="langchainQuickstart", metrics=[answer_relevancy_metric]
)

llm = OpenAI(
    temperature=0,
    callbacks=[deepeval_callback],
    verbose=True,
    openai_api_key="<YOUR_API_KEY>",
)

output = llm.generate(
    [
        "What is the best evaluation tool out there? (no bias at all)",
    ]
)

# 检查指标是否成功
print(answer_relevancy_metric.is_successful())  # 返回True/False

场景2：在不使用回调的情况下跟踪LLM

import requests
from langchain.chains import RetrievalQA
from langchain_chroma import Chroma
from langchain_community.document_loaders import TextLoader
from langchain_openai import OpenAI, OpenAIEmbeddings
from langchain_text_splitters import CharacterTextSplitter

# 使用API代理服务提高访问稳定性
text_file_url = "https://raw.githubusercontent.com/hwchase17/chat-your-data/master/state_of_the_union.txt"
openai_api_key = "sk-XXX"

with open("state_of_the_union.txt", "w") as f:
    response = requests.get(text_file_url)
    f.write(response.text)

loader = TextLoader("state_of_the_union.txt")
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key)
docsearch = Chroma.from_documents(texts, embeddings)

qa = RetrievalQA.from_chain_type(
    llm=OpenAI(openai_api_key=openai_api_key),
    chain_type="stuff",
    retriever=docsearch.as_retriever(),
)

query = "Who is the president?"
result = qa.run(query)

# 手动检查答案的相似性
answer_relevancy_metric.measure(result, query)
print(answer_relevancy_metric.is_successful())