前言
这回还是粗略写写前言吧,构建好RAG系统之后,你总得去进行测试吧,那么如何测试呢?用什么指标去衡量呢?测试数据集怎么构建呢?
这里使用Ragas对RAG系统进行测试,而Ragas又基本是OPENAI的接口,那是要钱钱的,所以就研究使用本地模型去跑
Ragas简介
不想写,有空再写
github地址:ragas
ragas测试用例数据集:
1、https://huggingface.co/datasets/explodinggradients/WikiEval
2、https://huggingface.co/datasets/explodinggradients/amnesty_qa
最好看看使用文档:
https://docs.ragas.io/en/stable/getstarted/index.html
下载Ragas
pip install ragas
然后就可以构建测试数据集啦,在构建前,你需要下载llama-index,因为我们需要基于llama-index去跑起本地的大模型和embedding模型
我的另外一篇博客:
LLM之基于llama-index部署本地embedding与GLM-4模型并初步搭建RAG(其他大模型也可)
搭建好之后,就可以直接构造数据集啦
构造测试数据集
from langchain_community.document_loaders import TextLoader
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
# from xinference.client import Client
from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
# from llama_index.core.embeddings import resolve_embed_model
#这个是使用llama_index 自定义的一个包,有看过我另外一篇博客的同学会明白的
from llama_index_self import GLMCustomLLM
#按文件夹输入
# loader = DirectoryLoader("/home/kelvin/nlp/GLM-4/basic_demo/input")
# documents = loader.load()
MODEL_PATH = '/home/kelvin/nlp/model/LLM/THUDM/glm-4-9b-chat'
embed_model_path = '/home/kelvin/nlp/model/Embedding/BAAI/bge-m3'
Settings.embed_model = HuggingFaceEmbedding(
model_name=f"{embed_model_path}",device='cuda'
)
Settings.llm = GLMCustomLLM(MODEL_PATH)
loader = TextLoader(file_path="/home/nlp/GLM-4/basic_demo/input/test.txt", encoding="utf-8")
documents_txt = loader.load()
generator = TestsetGenerator.from_llama_index(
Settings.llm,
Settings.llm,
Settings.embed_model
)
# generate testset
testset = generator.generate_with_langchain_docs(documents_txt, test_size=1024, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
testset.to_pandas()
testset.to_csv('./testset.csv', index=False)
testset.to_pickle('./testset.pkl')
testset.to_json('./testset.json')
print(f'finish')
官网构建测试数据集如下:
1、输入OPENAI_API_KEY
import os
os.environ["OPENAI_API_KEY"] = "your-openai-key"
2、读取某个文件夹,并读取文件夹下的文件
from langchain_community.document_loaders import DirectoryLoader
loader = DirectoryLoader("your-directory")
documents = loader.load()
3、构建测试数据集
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
# generator with openai models
generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
critic_llm = ChatOpenAI(model="gpt-4")
embeddings = OpenAIEmbeddings()
generator = TestsetGenerator.from_langchain(
generator_llm,
critic_llm,
embeddings
)
# generate testset
testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
欢迎大家点赞或收藏~
大家的点赞或收藏可以鼓励作者加快更新哟~