Milvus与M3E学习笔记

Milvus

报错信息

pip安装第三方库报错

1
yum install python-pip时，CentOS7默认的python版本是2.7。它不支持更新到最新版本。


yum install python3-pip
pip3 install --upgrade pip
yum remove python-pip


pip3 install numpy pymilvus


2
ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the ‘ssl’ module is compiled with OpenSSL 1.1.0h 27 Mar 2018.
https://blog.csdn.net/qq_42873925/article/details/131112721

pip install urllib3==1.26.15.


3、添加源
pip install numpy -i http://mirrors.aliyun.com/simple/ --trusted-host mirrors.aliyun.com
python -m pip install padas
pip install -U flask-cors
python -m pip install pymilvus --index-url https://pypi.tuna.tsinghua.edu.cn/simp

pip     install    numpy       -i    https://mirrors.aliyun.com/pypi/simple/
pip     install    pymilvus       -i    https://mirrors.aliyun.com/pypi/simple/
pip     install    sentence-transformers       -i    https://mirrors.aliyun.com/pypi/simple/
pip install urllib3==1.26.15 -i    https://mirrors.aliyun.com/pypi/simple/
pip     install    flask-cors       -i    https://mirrors.aliyun.com/pypi/simple/




其他源
1.1 清华大学（完全度和速度都很好，是一个优秀的pip镜像源）
https://pypi.tuna.tsinghua.edu.cn/simple
1.2 阿里云（完全度和速度也很好，是一个不错的选择）
https://mirrors.aliyun.com/pypi/simple/
1.3 网易（速度比较快，但是完全度有限）
https://mirrors.163.com/pypi/simple/ 
1.4 豆瓣（速度较快，但是完全度也有限）
https://pypi.douban.com/simple/ 
1.5 百度云（速度较快，但是完全度也有限）
https://mirror.baidu.com/pypi/simple/
1.6 中科大（速度较快，但完全度不如前面几个镜像源）
https://pypi.mirrors.ustc.edu.cn/simple/
1.7 华为云（完全度和速度均中等）
https://mirrors.huaweicloud.com/repository/pypi/simple/
1.8 腾讯云（速度一般，完全度也一般）
https://mirrors.cloud.tencent.com/pypi/simple/

等
https://zhuanlan.zhihu.com/p/623325525?utm_id=0

安装概念

官网
https://www.milvus-io.com/overview

https://github.com/milvus-io/milvus/releases

https://milvus.io/docs

安装
https://blog.csdn.net/sinat_39620217/article/details/131847096

学习连接
https://zhuanlan.zhihu.com/p/634255317

Helm部署Milvus集群
https://blog.csdn.net/weixin_48190891/article/details/123795134
http://www.manongjc.com/detail/29-yjtmwfcepsxqdmi.html
https://blog.csdn.net/qq_34939308/article/details/132162927


aliyun milvus

https://blog.csdn.net/qwe123321123/article/details/128700505

单节点部署

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.3.0
##旧版本     image: milvusdb/milvus:v2.2.11
    command: ["milvus", "run", "standalone"]
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"

  attu:
    container_name: attu
    image: zilliz/attu:v2.2.6
    environment:
      MILVUS_URL: milvus-standalone:19530
    ports:
      - "8000:3000"
    depends_on:
      - "standalone"


networks:
  default:
    name: milvus

M3e

介绍

https://blog.csdn.net/sinat_30045277/article/details/131208109
https://blog.csdn.net/fogdragon/article/details/132095193
https://huggingface.co/moka-ai/m3e-base

numpy pymilvus sentence-transformers

m3efalsk

https://blog.csdn.net/weixin_44455388/article/details/132738672


flask 接口
https://blog.csdn.net/mouday/article/details/122014783




form 测试

curl -XPOST -d "text=helloman" "http://172.26.21.93:5000/embeddings"

curl -XPOST -d "text=The CUDA Profiling Tools Interface (CUPTI) enables the creation of profiling and tracing tools that target CUDA applications. CUPTI provides the following APIs: the Activity API, the Callback API, the Event API, the Metric API, the Profiling API, the PC Sampling API and the Checkpoint API. Using these APIs, you can develop profiling tools that give insight into the CPU and GPU behavior of CUDA applications. CUPTI is delivered as a dynamic library on all platforms supported by CUDA.In this CUPTI document, Tracing refers to the collection of timestamps and additional information for CUDA activities such as CUDA APIs, kernel launches and memory copies during the execution of a CUDA application. Tracing helps in identifying performance issues for the CUDA code by telling you which parts of a program require the most time. Tracing information can be collected using the Activity and Callback APIs.In this CUPTI document, Profiling refers to the collection of GPU performance metrics for a single kernel or a set of kernels in isolation. Profiling might involve multiple replays of the kernel/s or the entire application to collect GPU performance metrics. For Volta and earlier GPU architectures, these metrics can be collected using CUPTI Event and Metric APIs. For Volta and later GPU architectures, the low overhead CUPTI Profiling and Perfworks Metric APIs replace this functionality, and a new CUPTI PC Sampling API is supported." "localhost:8888/embeddings"

SentenceTransformers-0002-概览-安装

https://zhuanlan.zhihu.com/p/643803077