在本教程中,我们将展示如何使用LlamaIndex与MongoDB创建一个文档存储和查询系统。通过将节点存储在文档存储中,我们可以在相同的基础文档存储上定义多个索引,而不是在索引之间复制数据。
环境设置
如果你在Colab上运行这个笔记本,你可能需要安装LlamaIndex。
!pip install llama-index-storage-docstore-mongodb
!pip install llama-index-storage-index-store-mongodb
!pip install llama-index-llms-openai
!pip install llama-index
import nest_asyncio
nest_asyncio.apply()
import logging
import sys
import os
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
导入必要的模块
from llama_index.core import SimpleDirectoryReader, StorageContext
from llama_index.core import VectorStoreIndex, SimpleKeywordTableIndex
from llama_index.core import SummaryIndex
from llama_index.core import ComposableGraph
from llama_index.llms.openai import OpenAI
from llama_index.core.response.notebook_utils import display_response
from llama_index.core import Settings
下载数据
!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'
加载文档
reader = SimpleDirectoryReader("./data/paul_graham/")
documents = reader.load_data()
解析为节点
from llama_index.core.node_parser import SentenceSplitter
nodes = SentenceSplitter().get_nodes_from_documents(documents)
添加到文档存储
MONGO_URI = os.environ["MONGO_URI"]
from llama_index.storage.docstore.mongodb import MongoDocumentStore
from llama_index.storage.index_store.mongodb import MongoIndexStore
storage_context = StorageContext.from_defaults(
docstore=MongoDocumentStore.from_uri(uri=MONGO_URI), # 中转API
index_store=MongoIndexStore.from_uri(uri=MONGO_URI), # 中转API
)
storage_context.docstore.add_documents(nodes)
定义多个索引
summary_index = SummaryIndex(nodes, storage_context=storage_context)
vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
keyword_table_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)
# 确认docstore中节点数量
len(storage_context.docstore.docs)
保存和加载测试
# 持久化存储上下文
storage_context.persist()
# 记录索引ID
list_id = summary_index.index_id
vector_id = vector_index.index_id
keyword_id = keyword_table_index.index_id
from llama_index.core import load_index_from_storage
# 重新创建存储上下文
storage_context = StorageContext.from_defaults(
docstore=MongoDocumentStore.from_uri(uri=MONGO_URI), # 中转API
index_store=MongoIndexStore.from_uri(uri=MONGO_URI), # 中转API
)
# 加载索引
summary_index = load_index_from_storage(
storage_context=storage_context, index_id=list_id
)
vector_index = load_index_from_storage(
storage_context=storage_context, vector_id=vector_id
)
keyword_table_index = load_index_from_storage(
storage_context=storage_context, keyword_id=keyword_id
)
测试一些查询
chatgpt = OpenAI(temperature=0, model="gpt-3.5-turbo")
Settings.llm = chatgpt
Settings.chunk_size = 1024
query_engine = summary_index.as_query_engine()
list_response = query_engine.query("What is a summary of this document?")
display_response(list_response)
query_engine = vector_index.as_query_engine()
vector_response = query_engine.query("What did the author do growing up?")
display_response(vector_response)
query_engine = keyword_table_index.as_query_engine()
keyword_response = query_engine.query("What did the author do after his time at YC?")
display_response(keyword_response)
可能遇到的错误
- 连接错误: 如果不能连接到MongoDB,请检查你的MongoDB URI是否正确,并确保MongoDB服务正在运行。
- 模块导入错误: 如果某些模块无法导入,请确保已经安装了所需的Python包。
如果你觉得这篇文章对你有帮助,请点赞,关注我的博客,谢谢!
参考资料: