背景
在不了解minio架构设计之前,我根据既往经验推测minio是将文件读写封装实现了一个应用系统,如要实现高可用,则需要部署两个minio节点,共享同一块磁盘。两个minio节组成一个集群,使用nginx实现负载均衡,这是一种常见的部署架构。
官方资料介绍的部署模式有两大类(https://min.io/docs/minio/linux/operations/installation.html),单节点部署和分布式部署,其中单节点又细分为单磁盘(只使用1块磁盘,不使用纠删码)和多磁盘(至少挂载4块磁盘,启用纠删码)。
在上述方案中,并不包含多节点共享同一块磁盘的模式,并且强调minio挂载的磁盘需要是干净的。
官方说明同时提到了站点复制模式,但该模式并不是共享磁盘,而是数据在多个站点同步(Data written to one site automatically replicates to the other peer site.)。
在了解了minio的文件高可用方案机制与原理后,还是有个疑惑,多节点共享磁盘到底是否可行?做了以下探索和验证。
共享同一块磁盘
首先尝试一种简单场景,minio部署两个节点,同享同一块磁盘。
docker run -p 10001:9000 --name minio-node1 -d --restart=always -e "MINIO_ACCESS_KEY=admin" -e "MINIO_SECRET_KEY=12345678" -v E:\dockerVolume\minio\data-share:/data minio/minio:RELEASE.2021-04-22T15-44-28Z server /data
docker run -p 10002:9000 --name minio-node2 -d --restart=always -e "MINIO_ACCESS_KEY=admin" -e "MINIO_SECRET_KEY=12345678" -v E:\dockerVolume\minio\data-share:/data minio/minio:RELEASE.2021-04-22T15-44-28Z server /data
两个节点均能启动成功。
启动日志显示正常,无警告更没有报错。
2023-11-29 11:12:09 Endpoint: http://172.17.0.3:9000 http://127.0.0.1:9000
2023-11-29 11:12:09
2023-11-29 11:12:09 Browser Access:
2023-11-29 11:12:09 http://172.17.0.3:9000 http://127.0.0.1:9000
2023-11-29 11:12:09
2023-11-29 11:12:09 Object API (Amazon S3 compatible):
2023-11-29 11:12:09 Go: https://docs.min.io/docs/golang-client-quickstart-guide
2023-11-29 11:12:09 Java: https://docs.min.io/docs/java-client-quickstart-guide
2023-11-29 11:12:09 Python: https://docs.min.io/docs/python-client-quickstart-guide
2023-11-29 11:12:09 JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
2023-11-29 11:12:09 .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
2023-11-29 11:12:09 IAM initialization complete
使用控制台创建桶及上传文件、下载文件、删除文件,从两个minio看上去,数据是同步的,都正常。
对容器进行任意重启、删除、新建操作,无报错、数据能正常读取。
共享多块磁盘
上面两个minio节点共享一块磁盘通过了测试,再往前迈一步,依旧是4个minio节点,然后每个minio节点挂载4块磁盘,这4块磁盘依旧是共享的,看看纠删码机制启用的情况下,共享磁盘的模式是否会产生相互干扰和数据错乱。
执行如下命令,创建容器
docker run -p 11001:9000 --name minio1 -d --restart=always -e "MINIO_ACCESS_KEY=admin" -e "MINIO_SECRET_KEY=12345678" -v E:\dockerVolume\minio\data1:/data1 -v E:\dockerVolume\minio\data2:/data2 -v E:\dockerVolume\minio\data3:/data3 -v E:\dockerVolume\minio\data4:/data4 minio/minio:RELEASE.2021-04-22T15-44-28Z server /data1 /data2 /data3 /data4
docker run -p 11002:9000 --name minio2 -d --restart=always -e "MINIO_ACCESS_KEY=admin" -e "MINIO_SECRET_KEY=12345678" -v E:\dockerVolume\minio\data1:/data1 -v E:\dockerVolume\minio\data2:/data2 -v E:\dockerVolume\minio\data3:/data3 -v E:\dockerVolume\minio\data4:/data4 minio/minio:RELEASE.2021-04-22T15-44-28Z server /data1 /data2 /data3 /data4
重复共享同一块磁盘的测试验证,结果依然正常。
非对称挂载磁盘
双节点,各挂载四块共享磁盘,仍旧正常运行。再进一步,磁盘不对称,推测应该会出问题。保持第一个节点挂载4块磁盘不变,先来尝试第二个节点挂载8块磁盘,是第1个节点的双倍关系,然后再尝试挂载6块磁盘。
docker run -p 21002:9000 --name minio2 -d --restart=always -e "MINIO_ACCESS_KEY=admin" -e "MINIO_SECRET_KEY=12345678" -v E:\dockerVolume\minio\data1:/data1 -v E:\dockerVolume\minio\data2:/data2 -v E:\dockerVolume\minio\data3:/data3 -v E:\dockerVolume\minio\data4:/data4 -v E:\dockerVolume\minio\data5:/data5 -v E:\dockerVolume\minio\data6:/data6 -v E:\dockerVolume\minio\data7:/data7 -v E:\dockerVolume\minio\data8:/data8 minio/minio:RELEASE.2021-04-22T15-44-28Z server /data1 /data2 /data3 /data4 /data5 /data6 /data7 /data8
直接启动报错,无限重启,提示data1磁盘已经被另外的纠删阵列使用了。
2023-11-29 14:03:52
2023-11-29 14:03:52 You are running an older version of MinIO released 2 years ago
2023-11-29 14:03:52 Update: Run `mc admin update`
2023-11-29 14:03:52
2023-11-29 14:03:52
2023-11-29 14:03:53 ERROR Unable to initialize backend: /data1 disk is already being used in another erasure deployment. (Number of disks specified: 8 but the number of disks found in the 1st disk's format.json: 4)
2023-11-29 14:03:55
2023-11-29 14:03:55 You are running an older version of MinIO released 2 years ago
2023-11-29 14:03:55 Update: Run `mc admin update`
2023-11-29 14:03:55
2023-11-29 14:03:55
2023-11-29 14:03:55 ERROR Unable to initialize backend: /data1 disk is already being used in another erasure deployment. (Number of disks specified: 8 but the number of disks found in the 1st disk's format.json: 4)
2023-11-29 14:03:57
2023-11-29 14:03:57 You are running an older version of MinIO released 2 years ago
2023-11-29 14:03:57 Update: Run `mc admin update`
2023-11-29 14:03:57
2023-11-29 14:03:57
2023-11-29 14:03:58 ERROR Unable to initialize backend: /data1 disk is already being used in another erasure deployment. (Number of disks specified: 8 but the number of disks found in the 1st disk's format.json: 4)
2023-11-29 14:04:01
2023-11-29 14:04:01 You are running an older version of MinIO released 2 years ago
2023-11-29 14:04:01 Update: Run `mc admin update`
2023-11-29 14:04:01
2023-11-29 14:04:01
2023-11-29 14:04:02 ERROR Unable to initialize backend: /data1 disk is already being used in another erasure deployment. (Number of disks specified: 8 but the number of disks found in the 1st disk's format.json: 4)
2023-11-29 14:04:06 ERROR Unable to initialize backend: /data1 disk is already being used in another erasure deployment. (Number of disks specified: 8 but the number of disks found in the 1st disk's format.json: 4)
2023-11-29 14:04:09
2023-11-29 14:04:09 You are running an older version of MinIO released 2 years ago
2023-11-29 14:04:09 Update: Run `mc admin update`
2023-11-29 14:04:09
2023-11-29 14:04:09
2023-11-29 14:04:10 ERROR Unable to initialize backend: /data1 disk is already being used in another erasure deployment. (Number of disks specified: 8 but the number of disks found in the 1st disk's format.json: 4)
挂载8块都出错了,后面就没必要验证挂载6块磁盘了。
结论
通过以上验证工作,基本符合原先的推测,即可以多个minio节点共享磁盘,将minio视为“应用”,将共享的磁盘视为“数据库”,几个minio节点组成集群,通过外置的负载均衡软件如Nginx实现高可用。
需要保持每个节点挂载磁盘的数量一致,可以只挂载一块磁盘,也可以挂载一组(>=4)磁盘,启用纠删码。
开源平台资料
平台名称:一二三开发平台
简介: 企业级通用开发平台
设计资料:csdn专栏
开源地址:Gitee
开源协议:MIT
欢迎收藏、点赞、评论,你的支持是我前行的动力。