本文将介绍如何使用Redis作为向量存储、缓存和文档存储,来构建一个摄取管道。我们将详细展示依赖安装、数据创建、运行摄取管道并确认文档是否成功摄取。
依赖
首先,我们需要安装Redis,并设置OpenAI API密钥。
%pip install llama-index-storage-docstore-redis
%pip install llama-index-vector-stores-redis
%pip install llama-index-embeddings-huggingface
!pip install redis
接着,启动Redis服务:
!docker run -d --name redis-stack -p 6379:6379 -p 8001:8001 redis/redis-stack:latest
配置OpenAI API密钥:
import os
os.environ["OPENAI_API_KEY"] = "sk-..."
创建初始数据
我们将创建一些测试数据:
!rm -rf test_redis_data
!mkdir -p test_redis_data
!echo "This is a test file: one!" > test_redis_data/test1.txt
!echo "This is a test file: two!" > test_redis_data/test2.txt
加载文档数据:
from llama_index.core import SimpleDirectoryReader
documents = SimpleDirectoryReader(
"./test_redis_data", filename_as_id=True
).load_data()
运行Redis摄取管道
我们将使用向量存储附加到管道,并处理数据的插入。
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core.ingestion import (
DocstoreStrategy,
IngestionPipeline,
IngestionCache,
)
from llama_index.core.ingestion.cache import RedisCache
from llama_index.storage.docstore.redis import RedisDocumentStore
from llama_index.core.node_parser import SentenceSplitter
from llama_index.vector_stores.redis import RedisVectorStore
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
pipeline = IngestionPipeline(
transformations=[
SentenceSplitter(),
embed_model,
],
docstore=RedisDocumentStore.from_host_and_port(
"localhost", 6379, namespace="document_store"
),
vector_store=RedisVectorStore(
index_name="redis_vector_store",
index_prefix="vectore_store",
redis_url="redis://localhost:6379",
),
cache=IngestionCache(
cache=RedisCache.from_host_and_port("localhost", 6379),
collection="redis_cache",
),
docstore_strategy=DocstoreStrategy.UPSERTS,
)
nodes = pipeline.run(documents=documents)
print(f"Ingested {len(nodes)} Nodes")
确认文档是否成功摄取
创建向量索引,并确认已摄取的文档。
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_vector_store(
pipeline.vector_store, embed_model=embed_model
)
print(
index.as_query_engine(similarity_top_k=10).query(
"What documents do you see?"
)
)
添加数据并重新摄取
更新已有文件并添加新文件:
!echo "This is a test file: three!" > test_redis_data/test3.txt
!echo "This is a NEW test file: one!" > test_redis_data/test1.txt
重新加载文档数据并运行管道:
documents = SimpleDirectoryReader(
"./test_redis_data", filename_as_id=True
).load_data()
nodes = pipeline.run(documents=documents)
print(f"Ingested {len(nodes)} Nodes")
确认文档:
index = VectorStoreIndex.from_vector_store(
pipeline.vector_store, embed_model=embed_model
)
response = index.as_query_engine(similarity_top_k=10).query(
"What documents do you see?"
)
print(response)
for node in response.source_nodes:
print(node.get_text())
可能遇到的错误
- Redis连接失败:确保Redis服务正在运行,并且地址和端口配置正确。
- API密钥问题:确认已正确设置OpenAI API密钥,且密钥有效。
- 依赖安装错误:检查依赖包是否安装成功,尤其是Redis和HuggingFace相关的库。
如果你觉得这篇文章对你有帮助,请点赞,关注我的博客,谢谢!