VDB之Chroma：Chroma/chromadb(一款优秀的向量数据库)的简介、安装、使用方法之详细攻略

一个处女座的程序猿

已于 2023-10-19 20:20:11 修改

阅读量1.1w

点赞数 9

分类专栏： NLP/LLMs BigData/Cloud Computing 文章标签：向量数据库自然语言处理基础大模型

于 2023-07-13 00:32:32 首次发布

本文链接：https://blog.csdn.net/qq_41185868/article/details/131692998

版权

NLP/LLMs 同时被 2 个专栏收录

765 篇文章

订阅专栏

BigData/Cloud Computing

60 篇文章

订阅专栏

VDB之Chroma：Chroma/chromadb(一款优秀的向量数据库)的简介、安装、使用方法之详细攻略

DB之VDB：向量数据库（Vector Database）的简介、常用库、使用方法之详细攻略

DB之VDB：向量数据库（Vector Database）的简介、常用库、使用方法之详细攻略

https://yunyaniu.blog.csdn.net/article/details/129106195

chroma的简介

2023年4月，Chroma获得1800万美元种子轮融资，除了机构投资者外，Chroma还获得了MongoDB、Scale、Hugging Face、Jasper等公司创始人或高管的投资，受到了整个生成式AI生态的欢迎。
Chroma是一个基于向量检索库实现的轻量级向量数据库，内置了入门所需的一切，并提供了简单的API。它目前只支持CPU计算，但可以利用乘积量化的方法，将一个向量的维度切成多段，每段分别进行k-means，从而减少存储空间和提高检索效率。它还可以与LangChain集成，实现基于语言模型的应用。

2023年6月正式发布Chroma，Chroma是一款AI 原生的开源嵌入数据库。Chroma 是一个开源的嵌入数据库。Chroma通过使知识、事实和技能可以插拔到 LLM（语言模型）中，从而使构建 LLM 应用变得简单。
Chroma 为您提供以下工具：存储嵌入及其元数据、嵌入文档和查询、搜索嵌入。Chroma 的优势在于简洁性和开发者生产力、在搜索之上的分析、它同时也非常快速。Chroma 包括 Python 客户端 SDK、JavaScript/TypeScript 客户端 SDK 和一个服务器应用程序。
在 Python 中，Chroma 可以在内存中运行，也可以在客户端/服务器（尚处于 alpha 阶段）模式下运行。在 JavaScript 中，Chroma 在客户端/服务器模式下运行，并与 Python 后端进行通信。
特点如下所示：
>> 轻松生成嵌入：Chroma 拥有您使用嵌入所需的所有工具
>> 简单易用：就像 pip install 一样，在笔记本中 5 秒内使用
>> 丰富的功能：搜索、过滤等等
>> 集成：直接插入 LangChain、LlamaIndex、OpenAI 等
>> JavaScript 客户端
>> 免费：Apache 2.0 开源许可
Chroma的优点是易用、轻量、智能，缺点是功能相对简单、不支持GPU加速。后续 Chroma 还会推出托管产品（Serverless 类产品），该产品将提供无服务器存储和检索功能，支持向上和向下扩展，让开发者开箱即用不需要自己搭建基础设施。

官网：Chroma

文档：🏡 Home | Chroma

GitHub：https://github.com/chroma-core/chroma

chroma的安装

pip install chromadb

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple chromadb


npm install --save chromadb # 或者 yarn add chromadb

C:\Windows\system32>pip install -i https://pypi.tuna.tsinghua.edu.cn/simple chromadb
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting chromadb
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/3c/ff/ac74735884031a3b9ddf7b1abecee0885ec61660588b1e7c6862bccf5116/chromadb-0.4.14-py3-none-any.whl (448 kB)
     |████████████████████████████████| 448 kB 726 kB/s
Requirement already satisfied: typing-extensions>=4.5.0 in d:\programdata\anaconda3\lib\site-packages (from chromadb) (4.8.0)
Collecting tqdm>=4.65.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/00/e5/f12a80907d0884e6dff9c16d0c0114d81b8cd07dc3ae54c5e962cc83037e/tqdm-4.66.1-py3-none-any.whl (78 kB)
     |████████████████████████████████| 78 kB 1.5 MB/s
Collecting grpcio>=1.58.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ed/b2/f37fa2dc8b9942c5d444adee073d683ff23a31a418214cc7d80f53f3285c/grpcio-1.59.0-cp39-cp39-win_amd64.whl (3.7 MB)
     |████████████████████████████████| 3.7 MB 142 kB/s
Collecting pulsar-client>=3.1.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/6f/13/b4b3f9282d274bacacf6268b946d00986ab14c35fe9f4113080bc9629ff8/pulsar_client-3.3.0-cp39-cp39-win_amd64.whl (3.4 MB)
     |████████████████████████████████| 3.4 MB 99 kB/s
Requirement already satisfied: onnxruntime>=1.14.1 in d:\programdata\anaconda3\lib\site-packages (from chromadb) (1.14.1)
Collecting chroma-hnswlib==0.7.3
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f0/f0/e197039fa81a122544fceccda4e6a2d08bdd4a70638f0a88b6b16dbc4adc/chroma_hnswlib-0.7.3-cp39-cp39-win_amd64.whl (150 kB)
     |████████████████████████████████| 150 kB 344 kB/s
Requirement already satisfied: tokenizers>=0.13.2 in d:\programdata\anaconda3\lib\site-packages (from chromadb) (0.13.3)
Collecting pypika>=0.48.9
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c7/2c/94ed7b91db81d61d7096ac8f2d325ec562fc75e35f3baea8749c85b28784/PyPika-0.48.9.tar.gz (67 kB)
     |████████████████████████████████| 67 kB 468 kB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting fastapi>=0.95.2
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/db/30/b8d323119c37e15b7fa639e65e0eb7d81eb675ba166ac83e695aad3bd321/fastapi-0.104.0-py3-none-any.whl (92 kB)
     |████████████████████████████████| 92 kB 122 kB/s
Collecting bcrypt>=4.0.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/46/81/d8c22cd7e5e1c6a7d48e41a1d1d46c92f17dae70a54d9814f746e6027dec/bcrypt-4.0.1-cp36-abi3-win_amd64.whl (152 kB)
     |████████████████████████████████| 152 kB 192 kB/s
Collecting typer>=0.9.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/bf/0e/c68adf10adda05f28a6ed7b9f4cd7b8e07f641b44af88ba72d9c89e4de7a/typer-0.9.0-py3-none-any.whl (45 kB)
     |████████████████████████████████| 45 kB 1.0 MB/s
Collecting overrides>=7.3.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/da/28/3fa6ef8297302fc7b3844980b6c5dbc71cdbd4b61e9b2591234214d5ab39/overrides-7.4.0-py3-none-any.whl (17 kB)
Collecting numpy>=1.22.5
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2d/ed/022fc4106f6d97e41e156201274138e0369b27dbfc8c206034f24ebd97d9/numpy-1.26.1-cp39-cp39-win_amd64.whl (15.8 MB)
     |████████████████████████████████| 15.8 MB 202 kB/s
Requirement already satisfied: pydantic>=1.9 in d:\programdata\anaconda3\lib\site-packages (from chromadb) (2.4.2)
Collecting requests>=2.28
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl (62 kB)
     |████████████████████████████████| 62 kB 1.5 MB/s
Collecting uvicorn[standard]>=0.18.3
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/79/96/b0882a1c3f7ef3dd86879e041212ae5b62b4bd352320889231cc735a8e8f/uvicorn-0.23.2-py3-none-any.whl (59 kB)
     |████████████████████████████████| 59 kB 3.3 MB/s
Collecting posthog>=2.4.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a7/73/35758818228c70348be4c3c66a76653c62e894e0e3c3461453c5341ca926/posthog-3.0.2-py2.py3-none-any.whl (37 kB)
Collecting importlib-resources
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/65/6e/09d8816b5cb7a4006ef8ad1717a2703ad9f331dae9717d9f22488a2d6469/importlib_resources-6.1.0-py3-none-any.whl (33 kB)
Collecting starlette<0.28.0,>=0.27.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/58/f8/e2cca22387965584a409795913b774235752be4176d276714e15e1a58884/starlette-0.27.0-py3-none-any.whl (66 kB)
     |████████████████████████████████| 66 kB 1.2 MB/s
Collecting anyio<4.0.0,>=3.7.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/19/24/44299477fe7dcc9cb58d0a57d5a7588d6af2ff403fdd2d47a246c91a3246/anyio-3.7.1-py3-none-any.whl (80 kB)
     |████████████████████████████████| 80 kB 1.7 MB/s
Requirement already satisfied: idna>=2.8 in d:\programdata\anaconda3\lib\site-packages (from anyio<4.0.0,>=3.7.1->fastapi>=0.95.2->chromadb) (3.3)
Collecting exceptiongroup
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ad/83/b71e58666f156a39fb29417e4c8ca4bc7400c0dd4ed9e8842ab54dc8c344/exceptiongroup-1.1.3-py3-none-any.whl (14 kB)
Requirement already satisfied: sniffio>=1.1 in d:\programdata\anaconda3\lib\site-packages (from anyio<4.0.0,>=3.7.1->fastapi>=0.95.2->chromadb) (1.2.0)
Requirement already satisfied: protobuf in d:\programdata\anaconda3\lib\site-packages (from onnxruntime>=1.14.1->chromadb) (3.19.6)
Requirement already satisfied: flatbuffers in d:\programdata\anaconda3\lib\site-packages (from onnxruntime>=1.14.1->chromadb) (2.0.7)
Requirement already satisfied: sympy in d:\programdata\anaconda3\lib\site-packages (from onnxruntime>=1.14.1->chromadb) (1.10.1)
Requirement already satisfied: packaging in d:\programdata\anaconda3\lib\site-packages (from onnxruntime>=1.14.1->chromadb) (21.3)
Requirement already satisfied: coloredlogs in d:\programdata\anaconda3\lib\site-packages (from onnxruntime>=1.14.1->chromadb) (15.0.1)
Requirement already satisfied: six>=1.5 in d:\programdata\anaconda3\lib\site-packages (from posthog>=2.4.0->chromadb) (1.16.0)
Collecting backoff>=1.10.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/df/73/b6e24bd22e6720ca8ee9a85a0c4a2971af8497d8f3193fa05390cbd46e09/backoff-2.2.1-py3-none-any.whl (15 kB)
Requirement already satisfied: monotonic>=1.5 in d:\programdata\anaconda3\lib\site-packages (from posthog>=2.4.0->chromadb) (1.5)
Requirement already satisfied: python-dateutil>2.1 in d:\programdata\anaconda3\lib\site-packages (from posthog>=2.4.0->chromadb) (2.8.2)
Requirement already satisfied: certifi in d:\programdata\anaconda3\lib\site-packages (from pulsar-client>=3.1.0->chromadb) (2021.10.8)
Requirement already satisfied: annotated-types>=0.4.0 in d:\programdata\anaconda3\lib\site-packages (from pydantic>=1.9->chromadb) (0.6.0)
Requirement already satisfied: pydantic-core==2.10.1 in d:\programdata\anaconda3\lib\site-packages (from pydantic>=1.9->chromadb) (2.10.1)
Requirement already satisfied: charset-normalizer<4,>=2 in d:\programdata\anaconda3\lib\site-packages (from requests>=2.28->chromadb) (2.0.12)
Requirement already satisfied: urllib3<3,>=1.21.1 in d:\programdata\anaconda3\lib\site-packages (from requests>=2.28->chromadb) (1.26.9)
Requirement already satisfied: colorama in d:\programdata\anaconda3\lib\site-packages (from tqdm>=4.65.0->chromadb) (0.4.4)
Requirement already satisfied: click<9.0.0,>=7.1.1 in d:\programdata\anaconda3\lib\site-packages (from typer>=0.9.0->chromadb) (8.0.4)
Collecting h11>=0.8
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/95/04/ff642e65ad6b90db43e668d70ffb6736436c7ce41fcc549f4e9472234127/h11-0.14.0-py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 1.6 MB/s
Collecting websockets>=10.4
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f4/3f/65dfa50084a06ab0a05f3ca74195c2c17a1c075b8361327d831ccce0a483/websockets-11.0.3-cp39-cp39-win_amd64.whl (124 kB)
     |████████████████████████████████| 124 kB 1.7 MB/s
Collecting httptools>=0.5.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/0a/0d/ca545a8a2831fc3e326fffecab268a2e7775e5ec4d57afc8f5ddc578cbd7/httptools-0.6.1-cp39-cp39-win_amd64.whl (60 kB)
     |████████████████████████████████| 60 kB 1.3 MB/s
Collecting python-dotenv>=0.13
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/44/2f/62ea1c8b593f4e093cc1a7768f0d46112107e790c3e478532329e434f00b/python_dotenv-1.0.0-py3-none-any.whl (19 kB)
Requirement already satisfied: pyyaml>=5.1 in d:\programdata\anaconda3\lib\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (6.0)
Collecting watchfiles>=0.13
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ef/c0/737ddb4c97efd7e4c98852973ca80d7fab65811d6c4d3a64182333d455b0/watchfiles-0.21.0-cp39-none-win_amd64.whl (280 kB)
     |████████████████████████████████| 280 kB 1.7 MB/s
Requirement already satisfied: humanfriendly>=9.1 in d:\programdata\anaconda3\lib\site-packages (from coloredlogs->onnxruntime>=1.14.1->chromadb) (10.0)
Requirement already satisfied: pyreadline3 in d:\programdata\anaconda3\lib\site-packages (from humanfriendly>=9.1->coloredlogs->onnxruntime>=1.14.1->chromadb) (3.4.1)
Requirement already satisfied: zipp>=3.1.0 in d:\programdata\anaconda3\lib\site-packages (from importlib-resources->chromadb) (3.7.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in d:\programdata\anaconda3\lib\site-packages (from packaging->onnxruntime>=1.14.1->chromadb) (3.0.4)
Requirement already satisfied: mpmath>=0.19 in d:\programdata\anaconda3\lib\site-packages (from sympy->onnxruntime>=1.14.1->chromadb) (1.2.1)
Building wheels for collected packages: pypika
  Building wheel for pypika (PEP 517) ... done
  Created wheel for pypika: filename=PyPika-0.48.9-py2.py3-none-any.whl size=53835 sha256=211a200d50e727e0f22f68bd12cee283f29544900120b047b61d3826d1424f27
  Stored in directory: c:\users\99386\appdata\local\pip\cache\wheels\3b\17\34\0b716a7c87f148d258492c55fe890d051f5ca7bcf9e045e582
Successfully built pypika
Installing collected packages: exceptiongroup, h11, anyio, websockets, watchfiles, uvicorn, starlette, requests, python-dotenv, numpy, httptools, backoff, typer, tqdm, pypika, pulsar-client, posthog, overrides, importlib-resources, grpcio, fastapi, chroma-hnswlib, bcrypt, chromadb
  Attempting uninstall: anyio
    Found existing installation: anyio 3.5.0
    Uninstalling anyio-3.5.0:
      Successfully uninstalled anyio-3.5.0
  Attempting uninstall: requests
    Found existing installation: requests 2.27.1
    Uninstalling requests-2.27.1:
      Successfully uninstalled requests-2.27.1
  Attempting uninstall: numpy
    Found existing installation: numpy 1.21.6
    Uninstalling numpy-1.21.6:
      Successfully uninstalled numpy-1.21.6
  Attempting uninstall: typer
    Found existing installation: typer 0.4.2
    Uninstalling typer-0.4.2:
      Successfully uninstalled typer-0.4.2
  Attempting uninstall: tqdm
    Found existing installation: tqdm 4.63.2
    Uninstalling tqdm-4.63.2:
      Successfully uninstalled tqdm-4.63.2
  Attempting uninstall: grpcio
    Found existing installation: grpcio 1.42.0
    Uninstalling grpcio-1.42.0:
      Successfully uninstalled grpcio-1.42.0
  Attempting uninstall: bcrypt
    Found existing installation: bcrypt 3.2.0
    Uninstalling bcrypt-3.2.0:
      Successfully uninstalled bcrypt-3.2.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
daal4py 2021.5.0 requires daal==2021.4.0, which is not installed.
conda-repo-cli 1.0.4 requires pathlib, which is not installed.
anaconda-project 0.10.2 requires ruamel-yaml, which is not installed.
xarray 2023.4.2 requires pandas>=1.4, but you have pandas 1.3.5 which is incompatible.
thinc 8.1.10 requires pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4, but you have pydantic 2.4.2 which is incompatible.
streamlit 1.24.0 requires protobuf<5,>=3.20, but you have protobuf 3.19.6 which is incompatible.
spacy 3.5.3 requires pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4, but you have pydantic 2.4.2 which is incompatible.
spacy 3.5.3 requires typer<0.8.0,>=0.3.0, but you have typer 0.9.0 which is incompatible.
scipy 1.8.0 requires numpy<1.25.0,>=1.17.3, but you have numpy 1.26.1 which is incompatible.
onnx 1.14.0 requires protobuf>=3.20.2, but you have protobuf 3.19.6 which is incompatible.
numba 0.55.1 requires numpy<1.22,>=1.18, but you have numpy 1.26.1 which is incompatible.
ludwig 0.7.4 requires transformers<4.22,>=4.10.1, but you have transformers 4.28.1 which is incompatible.
jupyter-server 1.13.5 requires pywinpty<2; os_name == "nt", but you have pywinpty 2.0.2 which is incompatible.
en-core-web-sm 3.0.0 requires spacy<3.1.0,>=3.0.0, but you have spacy 3.5.3 which is incompatible.
Successfully installed anyio-3.7.1 backoff-2.2.1 bcrypt-4.0.1 chroma-hnswlib-0.7.3 chromadb-0.4.14 exceptiongroup-1.1.3 fastapi-0.104.0 grpcio-1.59.0 h11-0.14.0 httptools-0.6.1 importlib-resources-6.1.0 numpy-1.26.1 overrides-7.4.0 posthog-3.0.2 pulsar-client-3.3.0 pypika-0.48.9 python-dotenv-1.0.0 requests-2.31.0 starlette-0.27.0 tqdm-4.66.1 typer-0.9.0 uvicorn-0.23.2 watchfiles-0.21.0 websockets-11.0.3

chroma的使用方法

1、基础用法

import chromadb
# setup Chroma in-memory, for easy prototyping. Can add persistence easily!
client = chromadb.Client()

# Create collection. get_collection, get_or_create_collection, delete_collection also available!
collection = client.create_collection("all-my-documents")

# Add docs to the collection. Can also update and delete. Row-based API coming soon!
collection.add(
    documents=["This is document1", "This is document2"], # we handle tokenization, embedding, and indexing automatically. You can skip that and add your own embeddings as well
    metadatas=[{"source": "notion"}, {"source": "google-docs"}], # filter on these!
    ids=["doc1", "doc2"], # unique for each doc
)

# Query/search 2 most similar results. You can also .get by id
results = collection.query(
    query_texts=["This is a query document"],
    n_results=2,
    # where={"metadata_field": "is_equal_to_this"}, # optional filter
    # where_document={"$contains":"search_string"}  # optional filter
)