Langchain使用之 - 文本分割Splitter

最新推荐文章于 2025-04-16 10:47:22 发布

taoli-qiao

最新推荐文章于 2025-04-16 10:47:22 发布

阅读量2w

点赞数 18

分类专栏： Langchain 文章标签： langchain

本文链接：https://blog.csdn.net/qiaotl/article/details/132715316

版权

Langchain提供了多种文本分割器，包括CharacterTextSplitter(),MarkdownHeaderTextSplitter(),RecursiveCharacterTextSplitter()等，各种Splitter的作用如下图所示：

TextSplitter

下面的代码是使用RecursiveCharacterTextSplitter对一段文字进行分割。

from langchain.text_splitter import RecursiveCharacterTextSplitter, CharacterTextSplitter

chunk_size = 20
chunk_overlap = 4

r_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap)
c_splitter = CharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap)

text = "hello world, how about you? thanks, I am fine.  the machine learning class. So what I wanna do today is just spend a little time going over the logistics of the class, and then