使用LlamaHub从OpenAI中下载和评估数据集

最新推荐文章于 2024-09-17 20:28:24 发布

qq_37836323

最新推荐文章于 2024-09-17 20:28:24 发布

阅读量285

点赞数 5

文章标签： python java 前端

本文链接：https://blog.csdn.net/qq_29929123/article/details/139798705

版权

使用LlamaHub从OpenAI中下载和评估数据集

本文将介绍如何使用LlamaHub下载数据集并使用自定义的RAG管道进行预测和评估。这对于需要检验模型性能的开发者非常有帮助。

安装依赖项

首先需要安装必要的依赖项：

!pip install llama-index-llms-openai

下载Llama数据集

我们将演示如何下载PaulGrahamEssayDataset数据集：

from llama_index.core.llama_dataset import download_llama_dataset

# 下载并安装依赖
rag_dataset, documents = download_llama_dataset(
    "PaulGrahamEssayDataset", "./paul_graham"
)

构建基础的RAG管道

使用下载的文档构建一个基础的RAG管道：

from llama_index.core import VectorStoreIndex

# 构建一个基础的RAG管道，使用默认设置
index = VectorStoreIndex.from_documents(documents=documents)
query_engine = index.as_query_engine()

手动进行预测

可以手动进行预测，也可以使用PredictAndEvaluatePack来完成一行代码的预测和评估：

import nest_asyncio
nest_asyncio.apply()

# 手动预测
prediction_dataset = await rag_dataset.amake_predictions_with(
    query_engine=query_engine, show_progress=True
)

进行评估

利用LLM作为评估者，进行响应和上下文的评价：

import tqdm
from llama_index.llms.openai import OpenAI
from llama_index.core.evaluation import (
    CorrectnessEvaluator,
    FaithfulnessEvaluator,
    RelevancyEvaluator,
    SemanticSimilarityEvaluator,
)

judges = {
    "correctness": CorrectnessEvaluator(llm=OpenAI(temperature=0, model="gpt-4")),
    "relevancy": RelevancyEvaluator(llm=OpenAI(temperature=0, model="gpt-4")),
    "faithfulness": FaithfulnessEvaluator(llm=OpenAI(temperature=0, model="gpt-4")),
    "semantic_similarity": SemanticSimilarityEvaluator()
}

evals = {
    "correctness": [],
    "relevancy": [],
    "faithfulness": [],
    "context_similarity": [],
}

for example, prediction in tqdm.tqdm(zip(rag_dataset.examples, prediction_dataset.predictions)):
    correctness_result = judges["correctness"].evaluate(
        query=example.query,
        response=prediction.response,
        reference=example.reference_answer,
    )
    relevancy_result = judges["relevancy"].evaluate(
        query=example.query,
        response=prediction.response,
        contexts=prediction.contexts,
    )
    faithfulness_result = judges["faithfulness"].evaluate(
        query=example.query,
        response=prediction.response,
        contexts=prediction.contexts,
    )
    semantic_similarity_result = judges["semantic_similarity"].evaluate(
        query=example.query,
        response="\n".join(prediction.contexts),
        reference="\n".join(example.reference_contexts),
    )
    evals["correctness"].append(correctness_result)
    evals["relevancy"].append(relevancy_result)
    evals["faithfulness"].append(faithfulness_result)
    evals["context_similarity"].append(semantic_similarity_result)

保存评估结果

将评估结果保存为JSON文件：

import json

# 保存评估结果
evaluations_objects = {
    "context_similarity": [e.dict() for e in evals["context_similarity"]],
    "correctness": [e.dict() for e in evals["correctness"]],
    "faithfulness": [e.dict() for e in evals["faithfulness"]],
    "relevancy": [e.dict() for e in evals["relevancy"]],
}

with open("evaluations.json", "w") as json_file:
    json.dump(evaluations_objects, json_file)

使用RagEvaluatorPack进行一键评估

使用RagEvaluatorPack进行一键评估：

from llama_index.core.llama_pack import download_llama_pack

RagEvaluatorPack = download_llama_pack("RagEvaluatorPack", "./pack")
rag_evaluator = RagEvaluatorPack(
    query_engine=query_engine, rag_dataset=rag_dataset, show_progress=True
)

benchmark_df = await rag_evaluator.runevaluate(
    batch_size=20,
    sleep_time_in_seconds=1,
)