MinIO 文件存储服务

MinIO 文件存储服务

2021-09-5

minio相对与fastdfs 来说，文档丰富，官方支持多种语言的SDK， 基于s3协议，至于具体存储性能上的差异，据说是优于fastdfs, 但本人没有做测试过比较。

官网文档：https://docs.min.io/

MinIO 安装部署

二进制包安装

二进制包下载(linux)

wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
# 简单启动服务
./minio server /data

其他不同CPU架构的安装包下载

Architecture	URL
64-bit Intel/AMD	https://dl.min.io/server/minio/release/linux-amd64/minio
64-bit ARM	https://dl.min.io/server/minio/release/linux-arm64/minio
64-bit PowerPC LE (ppc64le)	https://dl.min.io/server/minio/release/linux-ppc64le/minio
IBM Z-Series (S390X)	https://dl.min.io/server/minio/release/linux-s390x/minio

启动服务

# 默认API端口：9000
./minio server /data

出现以下警告

# 控制台端口为指定
WARNING: Console endpoint is listening on a dynamic port (39224), please use --console-address ":PORT" to choose a static port.
# 此警告表示使用的是默认的管理账号与密码
WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables


# 指定控制台端口
./minio server ./data --console-address ":9010"


启动成功后有如下显示
# http://200.200.200.160:9010 控制台页面

[root@localhost minio]# ./minio server ./data --console-address ":9010"
API: http://200.200.200.160:9000  http://127.0.0.1:9000       
RootUser: minioadmin 
RootPass: minioadmin 

Console: http://200.200.200.160:9010 http://127.0.0.1:9010    
RootUser: minioadmin 
RootPass: minioadmin 

Command-line: https://docs.min.io/docs/minio-client-quickstart-guide
   $ mc alias set myminio http://200.200.200.160:9000 minioadmin minioadmin

Documentation: https://docs.min.io
WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables

控制台使用（可视化管理）

新建Bucket

minio的数据都存储已在Bucket中， 当新建一个Bucket之后，就会在启动服务时指定的目录创建一个文件夹（与Bucket名称一致），然后可以在这个Bucket中上传文件。

上传文件

新建路径

客户端-MC

客户端MC安装

wget https://dl.min.io/client/mc/release/linux-amd64/mc

# 配置s3服务， myminio 是服务名， http://200.200.200.160:9000 是启动服务时的API
#  命令说明：mc alias set <ALIAS> <YOUR-S3-ENDPOINT> <YOUR-ACCESS-KEY> <YOUR-SECRET-KEY> --api <API-SIGNATURE> --path <BUCKET-LOOKUP-TYPE>

./mc alias set myminio http://200.200.200.160:9000 minioadmin minioadmin

查看客户端已经配置了哪些s3服务

[root@localhost ~]# mc alias list
gcs    
  URL       : https://storage.googleapis.com
  AccessKey : YOUR-ACCESS-KEY-HERE
  SecretKey : YOUR-SECRET-KEY-HERE
  API       : S3v2
  Path      : dns

local  
  URL       : http://localhost:9000
  AccessKey : 
  SecretKey : 
  API       : 
  Path      : auto

myminio # 这是刚才我们配置的
  URL       : http://200.200.200.160:9000
  AccessKey : minioadmin
  SecretKey : minioadmin
  API       : s3v4
  Path      : auto

更多命令

alias       set, remove and list aliases in configuration file
ls          list buckets and objects
mb          make a bucket
rb          remove a bucket
cp          copy objects
mirror      synchronize object(s) to a remote site
cat         display object contents
head        display first 'n' lines of an object
pipe        stream STDIN to an object
share       generate URL for temporary access to an object
find        search for objects
sql         run sql queries on objects
stat        show object metadata
mv          move objects
tree        list buckets and objects in a tree format
du          summarize disk usage recursively
retention   set retention for object(s)
legalhold   set legal hold for object(s)
diff        list differences in object name, size, and date between two buckets
rm          remove objects
encrypt    manage bucket encryption config
event       manage object notifications
watch       listen for object notification events
undo        undo PUT/DELETE operations
policy      manage anonymous access to buckets and objects
tag         manage tags for bucket(s) and object(s)
ilm         manage bucket lifecycle
version     manage bucket versioning
replicate   configure server side bucket replication
admin       manage MinIO servers
update      update mc to latest release

Python SDK

一个示例

# 这是官网的示例，直接copy，再做些必要改动
from minio import Minio
from minio.error import S3Error


def main():
    # Create a client with the MinIO server playground, its access key
    # and secret key.
    # 根据这个配置： ./mc alias set myminio http://200.200.200.160:9000 minioadmin minioadmin
    client = Minio(
        "200.200.200.160:9000",
        access_key="minioadmin",
        secret_key="minioadmin",
        secure=False, # 不使用HTTPS链接
    )
    print("====链接 client 成功======")
    # 判断是否存在bucket
    found = client.bucket_exists("test") # 这是之前创建的Bucket
    if not found:
        # 新建bucket  名为 test
        client.make_bucket("test")
    else:
        print("Bucket 'test' already exists")

    # 上传本地文件到指定bucket
    client.fput_object(
        "test", "test.txt", "./test.txt", # bucket 服务文件名， 本地路径
    )
    print(
        "./text.txt文件上传成功"
    )


if __name__ == "__main__":
    import traceback
    try:
        main()
    except S3Error as exc:
        print("error occurred.", traceback.format_exc())

Minio对象说明

from minio import Minio
client = Minio(endpoint, access_key=None, secret_key=None, session_token=None, secure=True, region=None, http_client=None, credentials=None)

# 官网说明， Minio建立的连接是线程安全的， 但是使用多进程共享一个Minio实例对象时是不安全的，所以每个进程需要新实例化一个Minio。
object is thread safe when using the Python threading library. Specifically, it is NOT safe to share it between multiple processes, for example when using multiprocessing.Pool. The solution is simply to create a new Minio object in each process, and not share it between processes.

Parameters

Param	Type	Description
endpoint	str	Hostname of a S3 service.
access_key	str	(Optional) Access key (aka user ID) of your account in S3 service.
secret_key	str	(Optional) Secret Key (aka password) of your account in S3 service.
session_token	str	(Optional) Session token of your account in S3 service.
secure	bool	(Optional) Flag to indicate to use secure (TLS) connection to S3 service or not.
region	str	(Optional) Region name of buckets in S3 service.
http_client	urllib3.poolmanager.PoolManager	(Optional) Customized HTTP client.
credentials	minio.credentials.Provider	(Optional) Credentials provider of your account in S3 service.

官网API连接：https://docs.min.io/docs/python-client-api-reference.html

SDK异常与报错

客户端与服务段时间不同步导致 RequestTimeTooSkewed

error occurred. Traceback (most recent call last): File “”, line 38, in main() File “”, line 18, in main found = client.bucket_exists(“test”) # 这是之前创建的Bucket File “c:\users\tesla\envs\ai\lib\site-packages\minio\api.py”, line 666, in bucket_exists self._execute(“HEAD”, bucket_name) File “c:\users\tesla\envs\ai\lib\site-packages\minio\api.py”, line 403, in _execute region = self._get_region(bucket_name, None) File “c:\users\tesla\envs\ai\lib\site-packages\minio\api.py”, line 475, in _get_region query_params={“location”: “”}, File “c:\users\tesla\envs\ai\lib\site-packages\minio\api.py”, line 389, in _url_open raise response_error minio.error.S3Error: S3 operation failed; code: RequestTimeTooSkewed, message: The difference between the request time and the server’s time is too large., resource: /test, request_id: , host_id: 1532d5b4-e32f-4a10-acea-115d48cc1ead

解决办法

同步时间即可

yum -y install ntp

ntpdate cn.pool.ntp.org