系统信息
OS Version
==============================================================
CentOS Linux release 7.6.1810 (Core)
MinIO Version
==============================================================
RELEASE.2021-08-17T20-53-08Z
Docker Version:
==============================================================
Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-208.git7d71120.el7_9.x86_64
Go version: go1.10.3
Git commit: 7d71120/1.13.1
Built: Mon Jun 7 15:36:09 2021
OS/Arch: linux/amd64
Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-208.git7d71120.el7_9.x86_64
Go version: go1.10.3
Git commit: 7d71120/1.13.1
Built: Mon Jun 7 15:36:09 2021
OS/Arch: linux/amd64
Experimental: false
目标
搭建两个集群,并且实现A集群下某个桶同步数据到B集群(单向同步,双向同步)
约定
* 所有可执行脚本开头都会加上(需要根据现场情况变更部分参数)
#!/bin/bash
#
# 2021/08/23
# ch
##############################
* 未增加则为参考脚本,需调整后执行
准备
测试服务器列表
名称 | ip | 备注 |
minio-cluster-1-node-1 | 192.168.2.159 | 集群A节点1 |
minio-cluster-1-node-2 | 192.168.2.160 | 集群A节点2 |
minio-cluster-2-node-1 | 192.168.2.168 | 集群B节点1 |
minio-cluster-2-node-2 | 192.168.2.158 | 集群B节点2 |
创建集群
修改Hosts文件
#!/bin/bash
#
# 2021/08/23
# ch
##############################
# 修改Hosts文件
# 根据实际情况选择对应节点
cat >> /etc/hosts << EOF
192.168.2.159 minio-cluster-1-node-1
192.168.2.160 minio-cluster-1-node-2
192.168.2.168 minio-cluster-2-node-1
192.168.2.158 minio-cluster-2-node-2
EOF
创建数据挂载目录
分别在4个机器上执行
#!/bin/bash
#
# 2021/08/23
# ch
##############################
rm -rf /home/export/minio
mkdir -p /home/export/minio/data1
mkdir -p /home/export/minio/data2
创建集群 minio-cluster-1
PS:以下三种方式等效
分别在 minio-cluster-1-node-1 和 minio-cluster-1-node-2 执行三种方式中的任意一种
第一种方式
修改Hosts文件并且利用Shell脚本语法
#!/bin/bash
#
# 2021/08/23
# ch
##############################
# 容器停止则立即删除容器 仅在测试环境中开启
RUN_MODEL="--rm"
# 生产环境请用 -d
# RUN_MODEL="-d"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="12345678"
echo 'RUN_MODEL='$RUN_MODEL
echo 'MINIO_ACCESS_KEY='$MINIO_ACCESS_KEY
echo 'MINIO_SECRET_KEY='$MINIO_SECRET_KEY
docker run --privileged=true $RUN_MODEL \
--net=host \
-e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \
-e MINIO_SECRET_KEY=$MINIO_SECRET_KEY \
-v /home/export/minio/data1:/data1 \
-v /home/export/minio/data2:/data2 \
minio/minio:RELEASE.2021-08-17T20-53-08Z server \
http://minio-cluster-1-node-{1...2}:9000/data{1...2}
第二种方式
修改Hosts文件
#!/bin/bash
#
# 2021/08/23
# ch
##############################
# 容器停止则立即删除容器 仅在测试环境中开启
START_MODEL="--rm"
# 生产环境请用 -d
# START_MODEL="-d"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="12345678"
echo 'RUN_MODEL='$RUN_MODEL
echo 'MINIO_ACCESS_KEY='$MINIO_ACCESS_KEY
echo 'MINIO_SECRET_KEY='$MINIO_SECRET_KEY
docker run --privileged=true $STOP_REMOVE \
--net=host \
-e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \
-e MINIO_SECRET_KEY=$MINIO_SECRET_KEY \
-v /home/export/minio/data1:/data1 \
-v /home/export/minio/data2:/data2 \
minio/minio:RELEASE.2021-08-17T20-53-08Z server \
http://minio-cluster-1-node-1:9000/data1 \
http://minio-cluster-1-node-1:9000/data2 \
http://minio-cluster-1-node-2:9000/data1 \
http://minio-cluster-1-node-2:9000/data2
第三种方式
啥都不用,纯硬写
#!/bin/bash
#
# 2021/08/23
# ch
##############################
# 容器停止则立即删除容器 仅在测试环境中开启
START_MODEL="--rm"
# 生产环境请用 -d
# START_MODEL="-d"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="12345678"
echo 'RUN_MODEL='$RUN_MODEL
echo 'MINIO_ACCESS_KEY='$MINIO_ACCESS_KEY
echo 'MINIO_SECRET_KEY='$MINIO_SECRET_KEY
docker run --privileged=true $STOP_REMOVE \
--net=host \
-e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \
-e MINIO_SECRET_KEY=$MINIO_SECRET_KEY \
-v /home/export/minio/data1:/data1 \
-v /home/export/minio/data2:/data2 \
minio/minio:RELEASE.2021-08-17T20-53-08Z server \
http://192.168.2.159:9000/data1 \
http://192.168.2.159:9000/data2 \
http://192.168.2.160:9000/data1 \
http://192.168.2.160:9000/data2
创建集群 minio-cluster-2
PS:以下三种方式等效
分别在 minio-cluster-2-node-1 和 minio-cluster-2-node-2 执行三种方式中的任意一种
第一种方式
修改Hosts文件并且利用Shell脚本语法
#!/bin/bash
#
# 2021/08/23
# ch
##############################
# 容器停止则立即删除容器 仅在测试环境中开启
RUN_MODEL="--rm"
# 生产环境请用 -d
# RUN_MODEL="-d"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="12345678"
echo 'RUN_MODEL='$RUN_MODEL
echo 'MINIO_ACCESS_KEY='$MINIO_ACCESS_KEY
echo 'MINIO_SECRET_KEY='$MINIO_SECRET_KEY
docker run --privileged=true $RUN_MODEL \
--net=host \
-e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \
-e MINIO_SECRET_KEY=$MINIO_SECRET_KEY \
-v /home/export/minio/data1:/data1 \
-v /home/export/minio/data2:/data2 \
minio/minio:RELEASE.2021-08-17T20-53-08Z server \
http://minio-cluster-2-node-{1...2}:9000/data{1...2}
第二种方式
修改Hosts文件
#!/bin/bash
#
# 2021/08/23
# ch
##############################
# 容器停止则立即删除容器 仅在测试环境中开启
START_MODEL="--rm"
# 生产环境请用 -d
# START_MODEL="-d"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="12345678"
echo 'RUN_MODEL='$RUN_MODEL
echo 'MINIO_ACCESS_KEY='$MINIO_ACCESS_KEY
echo 'MINIO_SECRET_KEY='$MINIO_SECRET_KEY
docker run --privileged=true $STOP_REMOVE \
--net=host \
-e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \
-e MINIO_SECRET_KEY=$MINIO_SECRET_KEY \
-v /home/export/minio/data1:/data1 \
-v /home/export/minio/data2:/data2 \
minio/minio:RELEASE.2021-08-17T20-53-08Z server \
http://minio-cluster-2-node-1:9000/data1 \
http://minio-cluster-2-node-1:9000/data2 \
http://minio-cluster-2-node-2:9000/data1 \
http://minio-cluster-2-node-2:9000/data2
第三种方式
啥都不用纯写
#!/bin/bash
#
# 2021/08/23
# ch
##############################
# 容器停止则立即删除容器 仅在测试环境中开启
START_MODEL="--rm"
# 生产环境请用 -d
# START_MODEL="-d"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="12345678"
echo 'RUN_MODEL='$RUN_MODEL
echo 'MINIO_ACCESS_KEY='$MINIO_ACCESS_KEY
echo 'MINIO_SECRET_KEY='$MINIO_SECRET_KEY
docker run --privileged=true $STOP_REMOVE \
--net=host \
-e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \
-e MINIO_SECRET_KEY=$MINIO_SECRET_KEY \
-v /home/export/minio/data1:/data1 \
-v /home/export/minio/data2:/data2 \
minio/minio:RELEASE.2021-08-17T20-53-08Z server \
http://192.168.2.168:9000/data1 \
http://192.168.2.168:9000/data2 \
http://192.168.2.158:9000/data1 \
http://192.168.2.158:9000/data2
安装MinIO客户端(MC)
PS:任意客户端(节点)安装即可
##############################
#
# 2021/08/23
# ch
##############################
mkdir -p /root/.bin
HOME="/root/.bin"
curl https://dl.min.io/client/mc/release/linux-amd64/mc \
--create-dirs \
-o $HOME/minio-binaries/mc
chmod +x $HOME/minio-binaries/mc
export PATH=$PATH:$HOME/minio-binaries/
mc --help
输出如下信息即安装成功
NAME:
mc - MinIO Client for cloud storage and filesystems.
USAGE:
mc [FLAGS] COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]
COMMANDS:
alias set, remove and list aliases in configuration file
ls list buckets and objects
mb make a bucket
rb remove a bucket
cp copy objects
mirror synchronize object(s) to a remote site
cat display object contents
head display first 'n' lines of an object
pipe stream STDIN to an object
share generate URL for temporary access to an object
find search for objects
sql run sql queries on objects
stat show object metadata
mv move objects
tree list buckets and objects in a tree format
du summarize disk usage recursively
retention set retention for object(s)
legalhold manage legal hold for object(s)
diff list differences in object name, size, and date between two buckets
rm remove objects
version manage bucket versioning
ilm manage bucket lifecycle
encrypt manage bucket encryption config
event manage object notifications
watch listen for object notification events
undo undo PUT/DELETE operations
anonymous manage anonymous access to buckets and objects
tag manage tags for bucket and object(s)
replicate configure server side bucket replication
admin manage MinIO servers
update update mc to latest release
GLOBAL FLAGS:
--autocompletion install auto-completion for your shell
--config-dir value, -C value path to configuration folder (default: "/root/.mc")
--quiet, -q disable progress bar display
--no-color disable color theme
--json enable JSON lines formatted output
--debug enable debug output
--insecure disable SSL certificate verification
--help, -h show help
--version, -v print the version
TIP:
Use 'mc --autocompletion' to enable shell autocompletion
VERSION:
RELEASE.2021-07-27T06-46-19Z
MC操作集群
添加远程集群
##############################
#
# 2021/08/23
# ch
##############################
# 集群多个节点取任意节点即可
mc alias set minio-cluster-1 http://192.168.2.159:9000 minio 12345678
mc alias set minio-cluster-2 http://192.168.2.168:9000 minio 12345678
验证
# 验证
mc alias list
看到如下输出即代表添加成功
minio-cluster-1
URL : http://192.168.2.159:9000
AccessKey : minio
SecretKey : 12345678
API : s3v4
Path : auto
minio-cluster-2
URL : http://192.168.2.168:9000
AccessKey : minio
SecretKey : 12345678
API : s3v4
Path : auto
mc admin info minio-cluster-1
看到如下输出即代表添加成功
● minio-cluster-1-node-2:9000
Uptime: 6 minutes
Version: 2021-08-17T20:53:08Z
Network: 2/2 OK
Drives: 2/2 OK
● minio-cluster-1-node-1:9000
Uptime: 7 minutes
Version: 2021-08-17T20:53:08Z
Network: 2/2 OK
Drives: 2/2 OK
4 drives online, 0 drives offline
mc admin info minio-cluster-2
看到如下输出即代表添加成功
● minio-cluster-2-node-2:9000
Uptime: 2 minutes
Version: 2021-08-17T20:53:08Z
Network: 2/2 OK
Drives: 2/2 OK
● minio-cluster-2-node-1:9000
Uptime: 2 minutes
Version: 2021-08-17T20:53:08Z
Network: 2/2 OK
Drives: 2/2 OK
4 drives online, 0 drives offline
创建桶
# 创建桶
# -p 存在则不执行,不存在则创建
mc mb -p minio-cluster-1/c1-n1-bk1
mc mb -p minio-cluster-2/c2-n2-bk1
# 验证
mc tree minio-cluster-1
# 看到如下信息则成功
minio-cluster-1
└─ c1-n1-bk1
mc tree minio-cluster-2
# 看到如下信息则成功
minio-cluster-2
└─ c2-n2-bk1
跨集群桶单向复制
启用桶版本控制
# 对上一步创建的桶开启版本控制
mc version enable minio-cluster-1/c1-n1-bk1
mc version enable minio-cluster-2/c2-n2-bk1
为目标集群创建复制目标
# 格式
mc admin bucket remote add AlphaReplication/SOURCEBUCKET \
http://bakerReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/DESTINATIONBUCKET
--service "replication"
[--sync]
# 可执行脚本
## 同步复制
mc admin bucket remote add minio-cluster-1/c1-n1-bk1 \
http://minio:12345678@minio-cluster-2-node-1:9000/c2-n2-bk1 \
--service "replication" \
--sync
## 异步复制
mc admin bucket remote add minio-cluster-1/c1-n1-bk1 \
http://minio:12345678@minio-cluster-2-node-1:9000/c2-n2-bk1
--service "replication"
# 验证
mc admin bucket remote ls minio-cluster-1
# 看到如下信息即代表成功
Remote URL Source ->Target ARN SYNC PROXY
http://minio-cluster-2-node-1:9000 c1-n1-bk1->c2-n2-bk1 arn:minio:replication::ff616aa8c6a08e40f654e716d963fe68e15d0a11941a586eb1cadcf0e076b2f8:c2-n2-bk1 sync
创建一个新的桶复制规则
# 格式
mc replicate add AlphaReplication/SOURCEBUCKET \
--remote-bucket DESTINATIONBUCKET \
--arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
--replicate "delete,delete-marker,existing-objects"
## 解释
# AlphaReplication/SOURCEBUCKET 对应源集群和桶
# arn 则是上一步创建复制规则返回的id
# --replicate "delete,delete-marker,existing-objects" 删除复制 以及复制现有对象
# 删除复制是指在源桶中删除一个对象回同步删除目标桶中的对象
# 可以不开
# 可执行脚本
mc replicate add minio-cluster-1/c1-n1-bk1 \
--remote-bucket c2-n2-bk1 \
--arn '' \
--replicate "delete,delete-marker,existing-objects"
验证桶复制
echo 'hello world' > index.html
mc cp ./index.html minio-cluster-1/c1-n1-bk1
mc ls minio-cluster-2/c2-n2-bk1
# 看到如下输出即代表成功
[2021-08-23 17:54:17 CST] 12B index.html
更为直接的验证方法是登陆
minio-cluster-1-node-1:9000 和 minio-cluster-2-node-1:9000
图形化页面进行验证
双向桶复制
双向复制增加目标
本质上在单向桶复制的基础上额外增加反向操作配置即可
增加 c-1-n1-bk1 -> c2-n2-bk1 同步目标
mc admin bucket remote add minio-cluster-1/c1-n1-bk1 \
http://minio:12345678@192.168.2.168:9000/c2-n2-bk1 \
--service "replication" \
--sync
增加 c2-n2-bk1 -> c-1-n1-bk1 同步目标
mc admin bucket remote add minio-cluster-2/c2-n2-bk1 \
http://minio:12345678@192.168.2.159:9000/c1-n1-bk1 \
--service "replication" \
--sync
双向复制增加规则
增加 c-1-n1-bk1 -> c2-n2-bk1 同步规则
mc replicate add minio-cluster-1/c1-n1-bk1 \
--remote-bucket c2-n2-bk1 \
--arn '' \
--replicate "delete,delete-marker,existing-objects"
增加 c2-n2-bk1 -> c-1-n1-bk1 同步规则
mc replicate add minio-cluster-2/c2-n2-bk1 \
--remote-bucket c1-n1-bk1 \
--arn 'arn:minio:replication::a379cafe5f496c3ba7fa95d5ff12a77c67ca3b10639a007ffd856bdb7fe3aa2f:c1-n1-bk1' \
--replicate "delete,delete-marker,existing-objects"
验证双向复制
echo 'hello world' > index.html
mc cp ./index.html minio-cluster-1/c1-n1-bk1
mc ls minio-cluster-2/c2-n2-bk1
# 看到如下输出即代表c1-n1-bk1 可以同步数据到 c2-n2-bk1
[2021-08-23 17:54:17 CST] 12B index.html
################### 分割线 #################################
echo 'hello world' > hellWord.txt
mc cp ./hellWord.txt minio-cluster-2/c2-n2-bk1
mc ls minio-cluster-1/c1-n1-bk1
# 看到如下输出即代表c1-n1-bk1 可以同步数据到 c2-n2-bk1
[2021-08-23 17:54:17 CST] 12B hellWord.txt
至此,Game Over !!!