LlamaIndex 相关性评估器使用指南
在大型语言模型(LLM)的应用中,确保生成的响应与查询相关且准确是至关重要的。LlamaIndex 提供了 RelevancyEvaluator
模块,帮助开发者评估响应和源节点是否与查询匹配。本文将详细介绍如何使用 RelevancyEvaluator
进行相关性评估。
安装依赖
首先,我们需要安装必要的依赖包:
%pip install llama-index-llms-openai pandas[jinja2] spacy
设置日志
为了更好地调试和查看运行过程中的信息,我们设置日志:
import logging
import sys
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
导入必要的模块
接下来,导入 LlamaIndex 和其他必要的模块:
from llama_index.core import (
TreeIndex,
VectorStoreIndex,
SimpleDirectoryReader,
Response,
)
from llama_index.llms.openai import OpenAI
from llama_index.core.evaluation import RelevancyEvaluator
from llama_index.core.node_parser import SentenceSplitter
import pandas as pd
pd.set_option("display.max_colwidth", 0)
使用 GPT-3 和 GPT-4 进行评估
我们使用 GPT-3 和 GPT-4 作为评估模型:
# gpt-3 (davinci)
gpt3 = OpenAI(temperature=0, model="gpt-3.5-turbo")
# gpt-4
gpt4 = OpenAI(temperature=0, model="gpt-4")
evaluator = RelevancyEvaluator(llm=gpt3)
evaluator_gpt4 = RelevancyEvaluator(llm=gpt4)
加载文档并创建向量索引
加载测试数据并创建向量索引:
documents = SimpleDirectoryReader("./test_wiki_data").load_data()
splitter = SentenceSplitter(chunk_size=512)
vector_index = VectorStoreIndex.from_documents(
documents, transformations=[splitter]
)
定义显示评估结果的函数
定义一个函数,用于在 Jupyter Notebook 中显示评估结果:
from llama_index.core.evaluation import EvaluationResult
def display_eval_df(
query: str, response: Response, eval_result: EvaluationResult
) -> None:
eval_df = pd.DataFrame(
{
"Query": query,
"Response": str(response),
"Source": response.source_nodes[0].node.text[:1000] + "...",
"Evaluation Result": "Pass" if eval_result.passing else "Fail",
"Reasoning": eval_result.feedback,
},
index=[0],
)
eval_df = eval_df.style.set_properties(
**{
"inline-size": "600px",
"overflow-wrap": "break-word",
},
subset=["Response", "Source"]
)
display(eval_df)
评估响应
运行查询并评估响应:
query_str = "What battles took place in New York City in the American Revolution?"
query_engine = vector_index.as_query_engine()
response_vector = query_engine.query(query_str)
eval_result = evaluator_gpt4.evaluate_response(
query=query_str, response=response_vector
)
display_eval_df(query_str, response_vector, eval_result)
query_str = "What are the airports in New York City?"
query_engine = vector_index.as_query_engine()
response_vector = query_engine.query(query_str)
eval_result = evaluator_gpt4.evaluate_response(
query=query_str, response=response_vector
)
display_eval_df(query_str, response_vector, eval_result)
query_str = "Who is the mayor of New York City?"
query_engine = vector_index.as_query_engine()
response_vector = query_engine.query(query_str)
eval_result = evaluator_gpt4.evaluate_response(
query=query_str, response=response_vector
)
display_eval_df(query_str, response_vector, eval_result)
评估源节点
评估返回的源节点,确定哪些源节点实际包含给定查询的答案:
from typing import List
def display_eval_sources(
query: str, response: Response, eval_result: List[str]
) -> None:
sources = [s.node.get_text() for s in response.source_nodes]
eval_df = pd.DataFrame(
{
"Source": sources,
"Eval Result": eval_result,
},
)
eval_df.style.set_caption(query)
eval_df = eval_df.style.set_properties(
**{
"inline-size": "600px",
"overflow-wrap": "break-word",
},
subset=["Source"]
)
display(eval_df)
query_str = "What are the airports in New York City?"
query_engine = vector_index.as_query_engine(
similarity_top_k=3, response_mode="no_text"
)
response_vector = query_engine.query(query_str)
eval_source_result_full = [
evaluator_gpt4.evaluate(
query=query_str,
response=response_vector.response,
contexts=[source_node.get_content()],
)
for source_node in response_vector.source_nodes
]
eval_source_result = [
"Pass" if result.passing else "Fail" for result in eval_source_result_full
]
display_eval_sources(query_str, response_vector, eval_source_result)
query_str = "Who is the mayor of New York City?"
query_engine = vector_index.as_query_engine(
similarity_top_k=3, response_mode="no_text"
)
eval_source_result_full = [
evaluator_gpt4.evaluate(
query=query_str,
response=response_vector.response,
contexts=[source_node.get_content()],
)
for source_node in response_vector.source_nodes
]
eval_source_result = [
"Pass" if result.passing else "Fail" for result in eval_source_result_full
]
display_eval_sources(query_str, response_vector, eval_source_result)
总结
通过 LlamaIndex 的 RelevancyEvaluator
模块,我们可以有效地评估查询引擎的响应和源节点是否与查询相关。这不仅有助于确保生成的内容准确无误,还能提升用户体验和系统的整体性能。希望本文能帮助你更好地理解和应用 LlamaIndex 的相关性评估功能。如果你有任何问题或需要进一步的帮助,请随时联系我们。祝你在 LLM 开发的道路上取得成功!