题意:如何持久化FAISS索引?
问题背景:
In the langchain wiki of FAISS, Faiss | 🦜️🔗 LangChain, it only talks about saving indexes to files.
在FAISS的LangChain维基页面上,它只讨论了将索引保存到文件中。
db.save_local("faiss_index")
new_db = FAISS.load_local("faiss_index", embeddings)
docs = new_db.similarity_search(query)
How can I save the indexes to databases, such that we can organize and concurrently access multiple indexes?
我如何能将索引保存到数据库中,以便我们可以组织和并发访问多个索引?
Searched online but could not get much info on this. Can FAISS be used with any kind of distributed databases?
我在网上搜索过,但关于这方面的信息不多。FAISS 可以与任何类型的分布式数据库一起使用吗?
问题解决:
In fact, FAISS is considered as an in-memory database itself in order to vector search based on similarity that you can serialize and deserialize the indexes using functions like write_index
and read_index
within the FAISS interface directly or using save_local
and load_local
within the LangChain integration which typically uses the pickle for serialization.
事实上,FAISS 本身被视为一种内存数据库,用于基于相似性的向量搜索。你可以使用 FAISS 接口内的 write_index
和 read_index
函数,或者直接在 LangChain 集成中使用 save_local
和 load_local
(后者通常使用 pickle 进行序列化)来序列化和反序列化索引。
If you need to store serialized files, you could manually save them in a NoSQL database like MongoDB as binary data, and then deserialize and retrieve them when needed, however, it is not the best practice!
如果你需要存储序列化后的文件,你可以手动将它们作为二进制数据保存在像 MongoDB 这样的 NoSQL 数据库中,然后在需要时再进行反序列化和检索。然而,这并不是最佳实践!
If you are looking for a vector database that is not in-memory and capable in a scalable system, you might want to consider using Milvus which is designed for this purpose.
如果你正在寻找一个不是内存内且能够在可扩展系统中使用的向量数据库,那么你可能想考虑使用 Milvus,它是为此目的而设计的。