MinIO 集群搭建以及跨集群桶单向/双向同步
Centos7.6
Minio
1-环境准备
集群一：4台服务器单硬盘
节点 数据目录（不能是/root下的） 进程/脚本目录
192.168.101.191 /data1/minio/data /opt/apps/minio
192.168.101.192 /data1/minio/data /opt/apps/minio
192.168.101.193 /data1/minio/data /opt/apps/minio
192.168.101.194 /data1/minio/data /opt/apps/minio
集群二：4台服务器单硬盘
节点 数据目录（不能是/root下的） 进程/脚本目录
192.168.101.196 /data1/minio/data /opt/apps/minio
192.168.101.197 /data1/minio/data /opt/apps/minio
192.168.101.198 /data1/minio/data /opt/apps/minio
192.168.101.199 /data1/minio/data /opt/apps/minio
时间保证同步
全部节点
[root@localhost ~]# date
#如时间不同更改
[root@localhost ~]# date -s 17:08
修改主机名及hosts文件
全部节点
#永久修改主机名
[root@localhost ~]# hostnamectl set-hostname sp-001
[root@localhost ~]# bash
#修改映射
[root@sd-001 ~]# vim /etc/hosts
#添加
192.168.156.2 sp-001
192.168.156.3 sp-002
192.168.156.4 sp-003
192.168.156.5 sp-004
准备安装包
#下载地址
wget https://dl.minio.io/server/minio/release/linux-amd64/minio
[root@sp-001 ~]# ll
-rw-r--r-- 1 root root 94044160 Nov 9 12:13 minio
2-添加新磁盘并挂载
列出所有可用块设备的信息
全部节点
[root@docker01 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdb 8:16 0 22G 0 disk #如刚添加的磁盘
新磁盘分区
全部节点
[root@docker01 ~]# fdisk /dev/sdb
命令(输入 m 获取帮助)：m #m
命令(输入 m 获取帮助)：n #n
Select (default p): p #p
分区号 (1-4，默认 1)：1 #1
起始 扇区 (2048-46137343，默认为 2048)： #回车
Last 扇区, +扇区 (2048-46137343，默认为 46137343)： #回车(全部分配)
命令(输入 m 获取帮助)：w #保存
查看磁盘信息
全部节点
[root@docker01 ~]# lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
sdb
└─sdb1 #已有sdb1分区
格式化磁盘
全部节点
[root@docker01 ~]# mkfs -t ext4 /dev/sdb1
挂载
全部节点
[root@docker01 ~]# mkdir /data1
[root@docker01 ~]# mount /dev/sdb2 /data1/
#查看挂载信息
[root@docker01 ~]# df -h
文件系统 容量 已用 可用 已用% 挂载点
/dev/sdb2 22G 45M 21G 1% /data1 #21G已挂载到/data1
开机自动挂载
全部节点
[root@minio-1 ~]# vi /etc/rc.d/rc.local
#添加：
mount /dev/vdb1 /data1/
给文件执行权限
[root@minio-1 ~]# chmod +x /etc/rc.d/rc.local
3-安装minio集群
配置存储桶复制的前提是 目标minio与源minio必须版本相同
3.1 安装minio集群1
创建安装目录
第一节点
[root@sp-001 ~]# mkdir -p /opt/apps/minio
移动安装包到安装目录及其他服务器
第一节点
[root@sp-001 ~]# mv minio /opt/apps/minio/
[root@sp-001 ~]# scp -r /opt/apps/minio/ 192.168.156.3:/opt/apps/
[root@sp-001 ~]# scp -r /opt/apps/minio/ 192.168.156.4:/opt/apps/
[root@sp-001 ~]# scp -r /opt/apps/minio/ 192.168.156.5:/opt/apps/
创建目录
全部节点
[root@sp-001 ~]# mkdir -p /data1/minio/data/ #数据目录
[root@sp-001 ~]# mkdir -p /data1/minio/logs/ #日志目录
[root@sp-001 ~]# mkdir -p /etc/minio #配置文件目录
集群启动文件说明
#1.Minio默认9000端口，在配置文件中加入–address “127.0.0.1:9029” 可更改端口
#2.MINIO_ACCESS_KEY：用户名，长度最小是5个字符
#3.MINIO_SECRET_KEY：密码，密码不能设置过于简单，不然minio会启动失败，长度最小是8个字符
#4.–config-dir：指定集群配置文件目录
节点1启动脚本
[root@sp-001 ~]# vim /opt/apps/minio/minio-run.sh
#新添
#!/bin/bash
export MINIO_ACCESS_KEY=mhsp
export MINIO_SECRET_KEY=mhsp@1234
/opt/apps/minio/minio server --config-dir /etc/minio \
--address "192.168.101.191:9000" \
http://192.168.101.191/data1/minio/data \
http://192.168.101.192/data1/minio/data \
http://192.168.101.193/data1/minio/data \
http://192.168.101.194/data1/minio/data > /data/minio/logs/minio.log
节点2启动脚本
[root@sp-002 ~]# vim /opt/apps/minio/minio-run.sh
#新添
#!/bin/bash
export MINIO_ACCESS_KEY=mhsp
export MINIO_SECRET_KEY=mhsp@1234
/opt/apps/minio/minio server --config-dir /etc/minio \
--address "192.168.101.192:9000" \
http://192.168.101.191/data1/minio/data \
http://192.168.101.192/data1/minio/data \
http://192.168.101.193/data1/minio/data \
http://192.168.101.194/data1/minio/data > /data/minio/logs/minio.log
节点3启动脚本
[root@sp-003 ~]# vim /opt/apps/minio/minio-run.sh
#新添
#!/bin/bash
export MINIO_ACCESS_KEY=mhsp
export MINIO_SECRET_KEY=mhsp@1234
/opt/apps/minio/minio server --config-dir /etc/minio \
--address "192.168.101.193:9000" \
http://192.168.101.191/data1/minio/data \
http://192.168.101.192/data1/minio/data \
http://192.168.101.193/data1/minio/data \
http://192.168.101.194/data1/minio/data > /data/minio/logs/minio.log
节点4启动脚本
[root@sp-004 ~]# vim /opt/apps/minio/minio-run.sh
#新添
#!/bin/bash
export MINIO_ACCESS_KEY=mhsp
export MINIO_SECRET_KEY=mhsp@1234
/opt/apps/minio/minio server --config-dir /etc/minio \
--address "192.168.101.194:9000" \
http://192.168.101.191/data1/minio/data \
http://192.168.101.192/data1/minio/data \
http://192.168.101.193/data1/minio/data \
http://192.168.101.194/data1/minio/data > /data/minio/logs/minio.log
3.2 安装minio集群2
集群2和集群1安装方式一样，注意配置文件IP更改即可
4-配置systemctl系统服务
创建Minio.server,将minio加入系统服务
全部节点
[root@sp-001 ~]# vim /etc/systemd/system/minio.service
#新添
[Unit]
Description=Minio service
Documentation=https://docs.minio.io/
[Service]
WorkingDirectory=/opt/apps/minio/
ExecStart=/opt/apps/minio/minio-run.sh
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
加载配置
全部节点
[root@sp-001 ~]# systemctl daemon-reload
权限修改
全部节点
[root@sp-001 ~]# chmod +x /opt/apps/minio/minio && chmod +x /opt/apps/minio/minio-run.sh
启动集群并开机自启
全部节点
[root@sp-001 ~]# systemctl start minio
[root@sp-001 ~]# systemctl enable minio
查看集群状态
root@sp-001 ~]# systemctl status minio.service -l
● minio.service - Minio service
Loaded: loaded (/etc/systemd/system/minio.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-11-23 21:32:06 CST; 14h ago
Docs: https://docs.minio.io/
Main PID: 19733 (minio-run.sh)
CGroup: /system.slice/minio.service
├─19733 /bin/bash /opt/apps/minio/minio-run.sh
└─19758 /opt/apps/minio/minio server --config-dir /etc/minio --address 192.168.156.2:9000 http://192.168.156.2/data/minio/data http://192.168.156.3/data/minio/data http://192.168.156.4/data/minio/data http://192.168.156.5/data/minio/data
Nov 24 sp-001 minio-run.sh[19733]: Error: Disk: http://192.168.156.3:9000/data/minio/data returned disk not found (*fmt.wrapError)
Nov sp-001 minio-run.sh[19733]: endpoint=http://192.168.156.3:9000/data/minio/data
Nov sp-001 minio-run.sh[19733]: 2: cmd/prepare-storage.go:52:cmd.glob..func7.1()
Nov sp-001 minio-run.sh[19733]: 1: cmd/erasure-sets.go:218:cmd.(*erasureSets).connectDisks.func1()
Nov sp-001 minio-run.sh[19733]: Client http://192.168.156.4:9000/minio/lock/v6 online
Nov sp-001 minio-run.sh[19733]: Client http://192.168.156.4:9000/minio/s.../data/v29 online
Nov sp-001 minio-run.sh[19733]: Client http://192.168.156.5:9000/minio/lock/v6 online
Nov sp-001 minio-run.sh[19733]: Client http://192.168.156.5:9000/minio/st.io/data/v29 online
Nov sp-001 minio-run.sh[19733]: Client http://192.168.156.3:9000/minio/storage.o/data/v29 online
Nov sp-001 minio-run.sh[19733]: Client http://192.168.156.3:9000/minio/lock/v6 online
5-登录页面测试
访问任意地址
http://192.168.156.2:9000
http://192.168.156.3:9000
http://192.168.156.4:9000
http://192.168.156.5:9000
6-安装MinIO客户端(MC)
任意客户端(节点)安装即可，此次在集群1节点1安装
创建存放目录
[root@hhht-1 ~]# mkdir -p /root/.bin
下载mc文件至指定目录
[root@hhht-1 ~]# curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o /root/.bin/minio-binaries/mc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 25.4M 100 25.4M 0 0 33288 0 0:13:20 0:13:20 --:--:-- 35158
#如无网环境在有网的电脑上下载mc文件，访问https://dl.min.io/client/mc/release/linux-amd64/mc
[root@hhht-1 ~]# cp mc /root/.bin/minio-binaries/mc
给文件赋权
[root@hhht-1 ~]# chmod +x /root/.bin/minio-binaries/mc
添加变量
[root@hhht-1 ~]# export PATH=$PATH:/root/.bin/minio-binaries/
测试mc是否安装成功
[root@hhht-1 ~]# mc --help
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── (q)uit/esc
quota manage bucket quota
encrypt manage bucket encryption config
event manage object notifications
watch listen for object notification events
undo undo PUT/DELETE operations
anonymous manage anonymous access to buckets and objects
tag manage tags for bucket and object(s)
diff list differences in object name, size, and date between two buckets
replicate configure server side bucket replication
admin manage MinIO servers
idp manage MinIO IDentity Provider server configuration
update update mc to latest release
ready checks if the cluster is ready or not
ping perform liveness check
od measure single stream upload and download
batch manage batch jobs
GLOBAL FLAGS:
--autocompletion install auto-completion for your shell
--config-dir value, -C value path to configuration folder (default: "/root/.mc") [$MC_CONFIG_DIR]
--quiet, -q disable progress bar display [$MC_QUIET]
--no-color disable color theme [$MC_NO_COLOR]
--json enable JSON lines formatted output [$MC_JSON]
--debug enable debug output [$MC_DEBUG]
--insecure disable SSL certificate verification [$MC_INSECURE]
--limit-upload value limits uploads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_UPLOAD]
--limit-download value limits downloads to a maximum rate in KiB/s, MiB/s, GiB/s. (default: unlimited) [$MC_LIMIT_DOWNLOAD]
--help, -h show help
--version, -v print the version
TIP:
Use 'mc --autocompletion' to enable shell autocompletion
COPYRIGHT:
Copyright (c) 2015-2023 MinIO, Inc.
LICENSE:
GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
输出如上信息即安装成功
7-利用桶复制实现minio双向同步
利用minio原生的存储桶复制实现2个集群之间指定bucket的双向同步
集群信息
#源minio集群：
192.168.101.191
192.168.101.192
192.168.101.193
192.168.101.194
目标minio集群：
192.168.101.196
192.168.101.197
192.168.101.198
192.168.101.199
添加Minio服务的访问权限，源端执行(置相关认证信息以别名设置)
集群多个节点取任意节点即可，以下操作全部在集群1节点1即可
#minio-cluster-1 # 集群1别名
#192.168.101.191:9000 # 源集群IP:端口
#admin admin_1357 # 集群1用户名密码
[root@hhht-1 ~]# mc alias set minio-cluster-1 http://192.168.101.191:9000 admin admin_1357
Added `minio-cluster-1` successfully.
#minio-cluster-1 # 集群2别名
#192.168.101.196:9000 # 源集群IP:端口
#admin admin_1357 # 集群2用户名密码
[root@hhht-1 ~]# mc alias set minio-cluster-2 http://192.168.101.196:9000 admin admin_1357
Added `minio-cluster-2` successfully.
验证，列出配置文件中的别名列表
看到如下输出即代表添加成功
[root@hhht-1 ~]# mc alias list
minio-cluster-1
URL : http://192.168.101.191:9000
AccessKey : admin
SecretKey : admin_1357
API : s3v4
Path : auto
minio-cluster-2
URL : http://192.168.101.196:9000
AccessKey : admin
SecretKey : admin_1357
API : s3v4
Path : auto
验证，显示 MinIO 服务器信息
看到如下输出即代表添加成功
[root@hhht-1 ~]# mc admin info minio-cluster-1
● 192.168.101.191:9000
Uptime: 20 hours
Version: 2023-11-01T18:37:25Z
Network: 4/4 OK
Drives: 1/1 OK
Pool: 1
● 192.168.101.192:9000
Uptime: 20 hours
Version: 2023-11-01T18:37:25Z
Network: 4/4 OK
Drives: 1/1 OK
Pool: 1
● 192.168.101.193:9000
Uptime: 20 hours
Version: 2023-11-01T18:37:25Z
Network: 4/4 OK
Drives: 1/1 OK
Pool: 1
● 192.168.101.194:9000
Uptime: 20 hours
Version: 2023-11-01T18:37:25Z
Network: 4/4 OK
Drives: 1/1 OK
Pool: 1
Pools:
1st, Erasure sets: 1, Drives per erasure set: 4
4 drives online, 0 drives offline
[root@hhht-1 ~]# mc admin info minio-cluster-2
● 192.168.101.196:9000
Uptime: 1 minute
Version: 2023-11-01T18:37:25Z
Network: 4/4 OK
Drives: 1/1 OK
Pool: 1
● 192.168.101.197:9000
Uptime: 57 seconds
Version: 2023-11-01T18:37:25Z
Network: 4/4 OK
Drives: 1/1 OK
Pool: 1
● 192.168.101.198:9000
Uptime: 47 seconds
Version: 2023-11-01T18:37:25Z
Network: 4/4 OK
Drives: 1/1 OK
Pool: 1
● 192.168.101.199:9000
Uptime: 41 seconds
Version: 2023-11-01T18:37:25Z
Network: 4/4 OK
Drives: 1/1 OK
Pool: 1
Pools:
1st, Erasure sets: 1, Drives per erasure set: 4
4 drives online, 0 drives offline
创建桶，源端执行
[root@hhht-1 ~]# mc mb -p minio-cluster-1/c1-n1-bk1
Bucket created successfully `minio-cluster-1/c1-n1-bk1`.
[root@hhht-1 ~]# mc mb -p minio-cluster-2/c2-n2-bk1
Bucket created successfully `minio-cluster-2/c2-n2-bk1`.
查看桶
看到如下信息则成功
[root@hhht-1 ~]# mc tree minio-cluster-1
minio-cluster-1
└─ c1-n1-bk1
[root@hhht-1 ~]# mc tree minio-cluster-2
minio-cluster-2
└─ c2-n2-bk1
启用桶版本控制
[root@hhht-1 ~]# mc version enable minio-cluster-1/c1-n1-bk1
[root@hhht-1 ~]# mc version enable minio-cluster-2/c2-n2-bk1
配置双向同步的主主复制，实现高可用，源端执行（注：若只是做灾备，配置单向同步就好了，不用2条都执行）
192.168.101.191-192.168.101.194集群复制到192.168.101.196-192.168.101.199集群
[root@hhht-1 ~]# mc replicate add --remote-bucket http://admin:admin_1357@192.168.101.196:9000/c2-n2-bk1 --replicate "delete,delete-marker,existing-objects" --sync minio-cluster-1/c1-n1-bk1 --limit-upload 200Mi
192.168.101.196-192.168.101.199集群复制到192.168.101.191-192.168.101.194集群
[root@hhht-1 ~]# mc replicate add --remote-bucket http://admin:admin_1357@192.168.101.191:9000/c1-n1-bk1 --replicate "delete,delete-marker,existing-objects" --sync minio-cluster-2/c2-n2-bk1 --limit-upload 200Mi
验证：源节点上传文件
[root@hhht-1 ~]# mc cp /etc/hosts minio-cluster-1/c1-n1-bk1
/etc/hosts: 374 B / 374 B ━━━━━━━━━━━━━━━━━━━━━━ 10.41 KiB/s 0s
验证：查看源节点文件是否存在
[root@hhht-1 ~]# mc ls minio-cluster-1/c1-n1-bk1
[2023-11-08 11:56:20 CST] 374B STANDARD hosts
验证：查看目标节点文件是否存在
[root@hhht-1 ~]# mc ls minio-cluster-2/c2-n2-bk1
[2023-11-08 11:56:20 CST] 374B STANDARD hosts
注：桶复制只对新文件做同步，存量数据建议 使用 rclone或mc 进行一次全量同步
集群1页面验证，在桶上传文件
>集群2查看文件
文件已同步至集群2
反之，在集群2上传文件，集群1查看文件也同步过来了，
也可以查看两个集群的minio数据目录(存储数据目录不能是/root目录，否则会报错)
#进入数据目录
[root@hhht-1 ~]# cd /data1/minio/data/
#刚才创建的c1-n1-bk1目录存在
[root@hhht-1 data]# ll
drwxr-xr-x 5 root root 4096 Nov 8 12:00 c1-n1-bk1
#进入目录,上传的文件存在
[root@hhht-1 data]# cd c1-n1-bk1/
[root@hhht-1 c1-n1-bk1]# ll
drwxr-xr-x 2 root root 4096 Nov 8 11:56 hosts
drwxr-xr-x 3 root root 4096 Nov 8 12:00 redis-3.0.0.gem
drwxr-xr-x 2 root root 4096 Nov 8 11:59 内蒙主机表.xlsx
查看其他服务器c1-n1-bk1目录及文件都存在
至此MinIO 集群搭建以及跨集群桶单向/双向同步部署完成
647
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
