MLFlow Tracking Server + MinIO + SQLite远程服务器配置方法

安装minio

docker pull minio/minio
docker run -d -p 9000:9000 -p 9090:9090 --name minio -e "MINIO_ACCESS_KEY=minio" -e "MINIO_SECRET_KEY=minio123" -v <本地目录>/data:/data -v <本地目录>/config:/root/.minio minio/minio server /data --console-address ":9000" --address ":9090"

• 这里注意一定要把9000 9090两个端口都映射出来，因为9000是登陆窗口，9090是api端口，如果没有映射9090或者在下面MLFLOW_S3_ENDPOINT_URL写错会出现botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the ListObjectsV2 operation: S3 API Requests must be made to API port.

（可选）安装SQLite

• 这一步主要是为了实现可选的model registry功能，支持的数据库有mysql, mssql, sqlite, postgresql，参数语法为<dialect>+<driver>://<username>:<password>@<host>:<port>/<database>，这里以最简单的sqlite为例

sudo apt install sqlite3
pip3 install --upgrade alembic

• 如果不执行更新alembic的话，会报错：

2023/03/07 11:32:07 ERROR mlflow.cli: Error initializing backend store
2023/03/07 11:32:07 ERROR mlflow.cli: Constraint must have a name
...
ValueError: Constraint must have a name

启动MLFLOW

pip3 install mlflow boto3
export AWS_ACCESS_KEY_ID=minio
export AWS_SECRET_ACCESS_KEY=minio123
export MLFLOW_S3_ENDPOINT_URL=http://0.0.0.0:9090
nohup mlflow server --backend-store-uri <本地目录>/mlflow --host 0.0.0.0 -p 5002 --backend-store-uri sqlite:///mlflow.db --default-artifact-root s3://mlflow > mlflow.log 2>&1 &

• 之后就可以在http://ip/5002访问MLFlow了

调用方法

• Python APIs: https://www.mlflow.org/docs/latest/python_api/index.html

# APIs: https://www.mlflow.org/docs/latest/python_api/index.html
import os
import mlflow
from mlflow import log_metric, log_param, log_artifacts
os.environ["AWS_ACCESS_KEY_ID"] = "minio"
os.environ["AWS_SECRET_ACCESS_KEY"] = "minio123"
os.environ["MLFLOW_S3_ENDPOINT_URL"] = f"{ip}:9090"
mlflow.set_tracking_uri(f"{ip}:5002")
mlflow.set_experiment("test")
log_param('lr', 0.001)
log_metric('loss', 0.01)
log_artifacts('./runs/exp1', , artifact_path="run")