[使用Comet追踪LangChain实验：从入门到实战]

bhawfgrcbtwny

于 2024-10-11 05:39:12 发布

阅读量179

点赞数 3

文章标签： langchain python 开发语言

本文链接：https://blog.csdn.net/bhawfgrcbtwny/article/details/142836743

版权

# 使用Comet追踪LangChain实验：从入门到实战

## 引言
在机器学习的世界中，管理和优化模型是至关重要的。Comet平台提供了一个集成的解决方案，可以帮助开发者管理、可视化和优化模型，从训练到生产监控。本文将演示如何将Comet用于跟踪Langchain实验，包括评估指标和LLM会话。

## 主要内容

### 安装Comet和依赖项
首先，确保你已经安装了`comet_ml`和其它所需的依赖项：

```bash
%pip install --upgrade --quiet comet_ml langchain langchain-openai google-search-results spacy textstat pandas
!{sys.executable} -m spacy download en_core_web_sm

初始化Comet并设置凭证

你可以在这里获取Comet API Key。初始化Comet后，设置OpenAI和SerpAPI的相关凭证：

import comet_ml
import os

comet_ml.init(project_name="comet-example-langchain")

os.environ["OPENAI_API_KEY"] = "..."  # 设置你的OpenAI API Key
os.environ["SERPAPI_API_KEY"] = "..."  # 设置你的SerpAPI API Key

场景1：使用LLM

在这个场景中，我们将使用一个语言模型（LLM）。通过Comet的回调功能，我们可以追踪模型的生成和性能指标。

from langchain_community.callbacks import CometCallbackHandler
from langchain_core.callbacks import StdOutCallbackHandler
from langchain_openai import OpenAI

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    complexity_metrics=True,
    stream_logs=True,
    tags=["llm"],
    visualizations=["dep"],
)
callbacks = [StdOutCallbackHandler(), comet_callback]
llm = OpenAI(temperature=0.9, callbacks=callbacks, verbose=True)

llm_result = llm.generate(["Tell me a joke", "Tell me a poem", "Tell me a fact"] * 3)
print("LLM result", llm_result)
comet_callback.flush_tracker(llm, finish=True)

场景2：在链中使用LLM

将LLM集成到链（Chain）中，并追踪其执行流程。

from langchain.chains import LLMChain
from langchain_core.prompts import PromptTemplate

template = """You are a playwright. Given the title of play, it is your job to write a synopsis for that title. 
Title: {title}
Playwright: This is a synopsis for the above play:"""
prompt_template = PromptTemplate(input_variables=["title"], template=template)
synopsis_chain = LLMChain(llm=llm, prompt=prompt_template, callbacks=callbacks)

test_prompts = [{"title": "Documentary about Bigfoot in Paris"}]
print(synopsis_chain.apply(test_prompts))
comet_callback.flush_tracker(synopsis_chain, finish=True)

场景3：使用代理和工具

在更复杂的场景中，集成代理和工具进行交互操作。

from langchain.agents import initialize_agent, load_tools

tools = load_tools(["serpapi", "llm-math"], llm=llm, callbacks=callbacks)
agent = initialize_agent(
    tools,
    llm,
    agent="zero-shot-react-description",
    callbacks=callbacks,
    verbose=True,
)
agent.run("Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?")
comet_callback.flush_tracker(agent, finish=True)

场景4：使用自定义评估指标

使用ROUGE指标来评估生成文本的质量。

from rouge_score import rouge_scorer

class Rouge:
    def __init__(self, reference):
        self.reference = reference
        self.scorer = rouge_scorer.RougeScorer(["rougeLsum"], use_stemmer=True)

    def compute_metric(self, generation, prompt_idx, gen_idx):
        prediction = generation.text
        results = self.scorer.score(target=self.reference, prediction=prediction)
        return {
            "rougeLsum_score": results["rougeLsum"].fmeasure,
            "reference": self.reference,
        }

reference = """
The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building.
It was the first structure to reach a height of 300 metres.
It is now taller than the Chrysler Building in New York City by 5.2 metres (17 ft)
Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France.
"""
rouge_score = Rouge(reference=reference)

template = """Given the following article, it is your job to write a summary.
Article: {article}
Summary: This is the summary for the above article:"""
prompt_template = PromptTemplate(input_variables=["article"], template=template)

synopsis_chain = LLMChain(llm=llm, prompt=prompt_template)
test_prompts = [{"article": "Some article text..."}]
print(synopsis_chain.apply(test_prompts, callbacks=callbacks))
comet_callback.flush_tracker(synopsis_chain, finish=True)