MinIO 集群搭建以及跨集群桶单向/双向同步

系统信息

OS Version
==============================================================
CentOS Linux release 7.6.1810 (Core)


MinIO Version
==============================================================
RELEASE.2021-08-17T20-53-08Z

Docker Version:
==============================================================
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-208.git7d71120.el7_9.x86_64
 Go version:      go1.10.3
 Git commit:      7d71120/1.13.1
 Built:           Mon Jun  7 15:36:09 2021
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-208.git7d71120.el7_9.x86_64
 Go version:      go1.10.3
 Git commit:      7d71120/1.13.1
 Built:           Mon Jun  7 15:36:09 2021
 OS/Arch:         linux/amd64
 Experimental:    false

目标

搭建两个集群,并且实现A集群下某个桶同步数据到B集群(单向同步,双向同步)

约定

* 所有可执行脚本开头都会加上(需要根据现场情况变更部分参数)

#!/bin/bash
#
# 2021/08/23
# ch
##############################

* 未增加则为参考脚本,需调整后执行


准备

测试服务器列表

名称

ip

备注

minio-cluster-1-node-1

192.168.2.159

集群A节点1

minio-cluster-1-node-2

192.168.2.160

集群A节点2

minio-cluster-2-node-1

192.168.2.168

集群B节点1

minio-cluster-2-node-2

192.168.2.158

集群B节点2

创建集群

修改Hosts文件

#!/bin/bash
#
# 2021/08/23
# ch
##############################
# 修改Hosts文件
# 根据实际情况选择对应节点
cat >> /etc/hosts << EOF
192.168.2.159 minio-cluster-1-node-1
192.168.2.160 minio-cluster-1-node-2
192.168.2.168 minio-cluster-2-node-1
192.168.2.158 minio-cluster-2-node-2
EOF

创建数据挂载目录

分别在4个机器上执行

#!/bin/bash
#
# 2021/08/23
# ch
##############################
rm -rf /home/export/minio
mkdir -p /home/export/minio/data1
mkdir -p /home/export/minio/data2

创建集群 minio-cluster-1

PS:以下三种方式等效

分别在 minio-cluster-1-node-1 和 minio-cluster-1-node-2 执行三种方式中的任意一种

第一种方式

修改Hosts文件并且利用Shell脚本语法

#!/bin/bash
#
# 2021/08/23
# ch
##############################

# 容器停止则立即删除容器 仅在测试环境中开启
RUN_MODEL="--rm"
# 生产环境请用 -d
# RUN_MODEL="-d"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="12345678"
echo 'RUN_MODEL='$RUN_MODEL
echo 'MINIO_ACCESS_KEY='$MINIO_ACCESS_KEY
echo 'MINIO_SECRET_KEY='$MINIO_SECRET_KEY

docker run --privileged=true $RUN_MODEL \
        --net=host \
        -e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \
        -e MINIO_SECRET_KEY=$MINIO_SECRET_KEY \
        -v /home/export/minio/data1:/data1 \
        -v /home/export/minio/data2:/data2 \
        minio/minio:RELEASE.2021-08-17T20-53-08Z server \
                http://minio-cluster-1-node-{1...2}:9000/data{1...2}

第二种方式

修改Hosts文件

#!/bin/bash
#
# 2021/08/23
# ch
##############################

# 容器停止则立即删除容器 仅在测试环境中开启
START_MODEL="--rm"
# 生产环境请用 -d
# START_MODEL="-d"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="12345678"
echo 'RUN_MODEL='$RUN_MODEL
echo 'MINIO_ACCESS_KEY='$MINIO_ACCESS_KEY
echo 'MINIO_SECRET_KEY='$MINIO_SECRET_KEY

docker run --privileged=true $STOP_REMOVE \
        --net=host \
        -e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \
        -e MINIO_SECRET_KEY=$MINIO_SECRET_KEY \
        -v /home/export/minio/data1:/data1 \
        -v /home/export/minio/data2:/data2 \
        minio/minio:RELEASE.2021-08-17T20-53-08Z server \
                http://minio-cluster-1-node-1:9000/data1 \
                http://minio-cluster-1-node-1:9000/data2 \
                http://minio-cluster-1-node-2:9000/data1 \
                http://minio-cluster-1-node-2:9000/data2

第三种方式

啥都不用,纯硬写

#!/bin/bash
#
# 2021/08/23
# ch
##############################

# 容器停止则立即删除容器 仅在测试环境中开启
START_MODEL="--rm"
# 生产环境请用 -d
# START_MODEL="-d"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="12345678"
echo 'RUN_MODEL='$RUN_MODEL
echo 'MINIO_ACCESS_KEY='$MINIO_ACCESS_KEY
echo 'MINIO_SECRET_KEY='$MINIO_SECRET_KEY

docker run --privileged=true $STOP_REMOVE \
        --net=host \
        -e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \
        -e MINIO_SECRET_KEY=$MINIO_SECRET_KEY \
        -v /home/export/minio/data1:/data1 \
        -v /home/export/minio/data2:/data2 \
        minio/minio:RELEASE.2021-08-17T20-53-08Z server \
                http://192.168.2.159:9000/data1 \
                http://192.168.2.159:9000/data2 \
                http://192.168.2.160:9000/data1 \
                http://192.168.2.160:9000/data2

创建集群 minio-cluster-2

PS:以下三种方式等效

分别在 minio-cluster-2-node-1 和 minio-cluster-2-node-2 执行三种方式中的任意一种

第一种方式

修改Hosts文件并且利用Shell脚本语法

#!/bin/bash
#
# 2021/08/23
# ch
##############################

# 容器停止则立即删除容器 仅在测试环境中开启
RUN_MODEL="--rm"
# 生产环境请用 -d
# RUN_MODEL="-d"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="12345678"
echo 'RUN_MODEL='$RUN_MODEL
echo 'MINIO_ACCESS_KEY='$MINIO_ACCESS_KEY
echo 'MINIO_SECRET_KEY='$MINIO_SECRET_KEY

docker run --privileged=true $RUN_MODEL \
        --net=host \
        -e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \
        -e MINIO_SECRET_KEY=$MINIO_SECRET_KEY \
        -v /home/export/minio/data1:/data1 \
        -v /home/export/minio/data2:/data2 \
        minio/minio:RELEASE.2021-08-17T20-53-08Z server \
                http://minio-cluster-2-node-{1...2}:9000/data{1...2}

第二种方式

修改Hosts文件

#!/bin/bash
#
# 2021/08/23
# ch
##############################

# 容器停止则立即删除容器 仅在测试环境中开启
START_MODEL="--rm"
# 生产环境请用 -d
# START_MODEL="-d"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="12345678"
echo 'RUN_MODEL='$RUN_MODEL
echo 'MINIO_ACCESS_KEY='$MINIO_ACCESS_KEY
echo 'MINIO_SECRET_KEY='$MINIO_SECRET_KEY

docker run --privileged=true $STOP_REMOVE \
        --net=host \
        -e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \
        -e MINIO_SECRET_KEY=$MINIO_SECRET_KEY \
        -v /home/export/minio/data1:/data1 \
        -v /home/export/minio/data2:/data2 \
        minio/minio:RELEASE.2021-08-17T20-53-08Z server \
                http://minio-cluster-2-node-1:9000/data1 \
                http://minio-cluster-2-node-1:9000/data2 \
                http://minio-cluster-2-node-2:9000/data1 \
                http://minio-cluster-2-node-2:9000/data2

第三种方式

啥都不用纯写

#!/bin/bash
#
# 2021/08/23
# ch
##############################

# 容器停止则立即删除容器 仅在测试环境中开启
START_MODEL="--rm"
# 生产环境请用 -d
# START_MODEL="-d"
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="12345678"
echo 'RUN_MODEL='$RUN_MODEL
echo 'MINIO_ACCESS_KEY='$MINIO_ACCESS_KEY
echo 'MINIO_SECRET_KEY='$MINIO_SECRET_KEY

docker run --privileged=true $STOP_REMOVE \
        --net=host \
        -e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \
        -e MINIO_SECRET_KEY=$MINIO_SECRET_KEY \
        -v /home/export/minio/data1:/data1 \
        -v /home/export/minio/data2:/data2 \
        minio/minio:RELEASE.2021-08-17T20-53-08Z server \
                http://192.168.2.168:9000/data1 \
                http://192.168.2.168:9000/data2 \
                http://192.168.2.158:9000/data1 \
                http://192.168.2.158:9000/data2

安装MinIO客户端(MC)

PS:任意客户端(节点)安装即可

##############################
#
# 2021/08/23
# ch
##############################
mkdir -p /root/.bin
HOME="/root/.bin"
curl https://dl.min.io/client/mc/release/linux-amd64/mc \
  --create-dirs \
  -o $HOME/minio-binaries/mc

chmod +x $HOME/minio-binaries/mc
export PATH=$PATH:$HOME/minio-binaries/

mc --help

输出如下信息即安装成功

NAME:
  mc - MinIO Client for cloud storage and filesystems.

USAGE:
  mc [FLAGS] COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]

COMMANDS:
  alias      set, remove and list aliases in configuration file
  ls         list buckets and objects
  mb         make a bucket
  rb         remove a bucket
  cp         copy objects
  mirror     synchronize object(s) to a remote site
  cat        display object contents
  head       display first 'n' lines of an object
  pipe       stream STDIN to an object
  share      generate URL for temporary access to an object
  find       search for objects
  sql        run sql queries on objects
  stat       show object metadata
  mv         move objects
  tree       list buckets and objects in a tree format
  du         summarize disk usage recursively
  retention  set retention for object(s)
  legalhold  manage legal hold for object(s)
  diff       list differences in object name, size, and date between two buckets
  rm         remove objects
  version    manage bucket versioning
  ilm        manage bucket lifecycle
  encrypt    manage bucket encryption config
  event      manage object notifications
  watch      listen for object notification events
  undo       undo PUT/DELETE operations
  anonymous  manage anonymous access to buckets and objects
  tag        manage tags for bucket and object(s)
  replicate  configure server side bucket replication
  admin      manage MinIO servers
  update     update mc to latest release

GLOBAL FLAGS:
  --autocompletion              install auto-completion for your shell
  --config-dir value, -C value  path to configuration folder (default: "/root/.mc")
  --quiet, -q                   disable progress bar display
  --no-color                    disable color theme
  --json                        enable JSON lines formatted output
  --debug                       enable debug output
  --insecure                    disable SSL certificate verification
  --help, -h                    show help
  --version, -v                 print the version

TIP:
  Use 'mc --autocompletion' to enable shell autocompletion

VERSION:
  RELEASE.2021-07-27T06-46-19Z

MC操作集群

添加远程集群

##############################
#
# 2021/08/23
# ch
##############################
# 集群多个节点取任意节点即可
mc alias set minio-cluster-1 http://192.168.2.159:9000 minio 12345678
mc alias set minio-cluster-2 http://192.168.2.168:9000 minio 12345678

验证

# 验证
mc alias list

看到如下输出即代表添加成功

minio-cluster-1
  URL       : http://192.168.2.159:9000
  AccessKey : minio
  SecretKey : 12345678
  API       : s3v4
  Path      : auto

minio-cluster-2
  URL       : http://192.168.2.168:9000
  AccessKey : minio
  SecretKey : 12345678
  API       : s3v4
  Path      : auto
mc admin info minio-cluster-1

看到如下输出即代表添加成功

●  minio-cluster-1-node-2:9000
   Uptime: 6 minutes
   Version: 2021-08-17T20:53:08Z
   Network: 2/2 OK
   Drives: 2/2 OK

●  minio-cluster-1-node-1:9000
   Uptime: 7 minutes
   Version: 2021-08-17T20:53:08Z
   Network: 2/2 OK
   Drives: 2/2 OK

4 drives online, 0 drives offline


mc admin info minio-cluster-2

看到如下输出即代表添加成功

●  minio-cluster-2-node-2:9000
   Uptime: 2 minutes
   Version: 2021-08-17T20:53:08Z
   Network: 2/2 OK
   Drives: 2/2 OK

●  minio-cluster-2-node-1:9000
   Uptime: 2 minutes
   Version: 2021-08-17T20:53:08Z
   Network: 2/2 OK
   Drives: 2/2 OK

4 drives online, 0 drives offline

创建桶

# 创建桶
# -p 存在则不执行,不存在则创建
mc mb -p minio-cluster-1/c1-n1-bk1

mc mb -p minio-cluster-2/c2-n2-bk1

# 验证
mc tree minio-cluster-1
# 看到如下信息则成功

minio-cluster-1
└─ c1-n1-bk1

mc tree minio-cluster-2
# 看到如下信息则成功

minio-cluster-2
└─ c2-n2-bk1

跨集群桶单向复制

启用桶版本控制

# 对上一步创建的桶开启版本控制
mc version enable minio-cluster-1/c1-n1-bk1
mc version enable minio-cluster-2/c2-n2-bk1

为目标集群创建复制目标

# 格式
mc admin bucket remote add AlphaReplication/SOURCEBUCKET \
   http://bakerReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/DESTINATIONBUCKET
   --service "replication"
   [--sync]
   
# 可执行脚本
## 同步复制
mc admin bucket remote add minio-cluster-1/c1-n1-bk1 \
   http://minio:12345678@minio-cluster-2-node-1:9000/c2-n2-bk1 \
   --service "replication" \
   --sync
   
## 异步复制
mc admin bucket remote add minio-cluster-1/c1-n1-bk1 \
   http://minio:12345678@minio-cluster-2-node-1:9000/c2-n2-bk1
   --service "replication"
   
   
# 验证
mc admin bucket remote ls minio-cluster-1
# 看到如下信息即代表成功
Remote URL                          Source   ->Target    ARN                                                                                               SYNC PROXY
http://minio-cluster-2-node-1:9000  c1-n1-bk1->c2-n2-bk1 arn:minio:replication::ff616aa8c6a08e40f654e716d963fe68e15d0a11941a586eb1cadcf0e076b2f8:c2-n2-bk1 sync

创建一个新的桶复制规则

# 格式
mc replicate add AlphaReplication/SOURCEBUCKET \
   --remote-bucket DESTINATIONBUCKET \
   --arn 'arn:minio:replication::<UUID>:DESTINATIONBUCKET' \
   --replicate "delete,delete-marker,existing-objects"
## 解释
# AlphaReplication/SOURCEBUCKET 对应源集群和桶
# arn 则是上一步创建复制规则返回的id
# --replicate "delete,delete-marker,existing-objects" 删除复制 以及复制现有对象
# 删除复制是指在源桶中删除一个对象回同步删除目标桶中的对象
# 可以不开

# 可执行脚本
mc replicate add minio-cluster-1/c1-n1-bk1 \
   --remote-bucket c2-n2-bk1 \
   --arn '' \
   --replicate "delete,delete-marker,existing-objects"

验证桶复制

echo 'hello world' > index.html
mc cp ./index.html minio-cluster-1/c1-n1-bk1
mc ls minio-cluster-2/c2-n2-bk1

# 看到如下输出即代表成功
[2021-08-23 17:54:17 CST]    12B index.html

更为直接的验证方法是登陆

minio-cluster-1-node-1:9000 和 minio-cluster-2-node-1:9000

图形化页面进行验证

双向桶复制

双向复制增加目标

本质上在单向桶复制的基础上额外增加反向操作配置即可

增加 c-1-n1-bk1 -> c2-n2-bk1 同步目标

mc admin bucket remote add minio-cluster-1/c1-n1-bk1 \
    http://minio:12345678@192.168.2.168:9000/c2-n2-bk1 \
    --service "replication" \
    --sync

增加 c2-n2-bk1 -> c-1-n1-bk1 同步目标

mc admin bucket remote add minio-cluster-2/c2-n2-bk1 \
    http://minio:12345678@192.168.2.159:9000/c1-n1-bk1 \
    --service "replication" \
    --sync

双向复制增加规则

增加 c-1-n1-bk1 -> c2-n2-bk1 同步规则

mc replicate add minio-cluster-1/c1-n1-bk1 \
    --remote-bucket c2-n2-bk1 \
    --arn '' \
    --replicate "delete,delete-marker,existing-objects"

增加 c2-n2-bk1 -> c-1-n1-bk1 同步规则

mc replicate add minio-cluster-2/c2-n2-bk1 \
    --remote-bucket c1-n1-bk1 \
    --arn 'arn:minio:replication::a379cafe5f496c3ba7fa95d5ff12a77c67ca3b10639a007ffd856bdb7fe3aa2f:c1-n1-bk1' \
    --replicate "delete,delete-marker,existing-objects"

验证双向复制

echo 'hello world' > index.html
mc cp ./index.html minio-cluster-1/c1-n1-bk1
mc ls minio-cluster-2/c2-n2-bk1

# 看到如下输出即代表c1-n1-bk1 可以同步数据到 c2-n2-bk1
[2021-08-23 17:54:17 CST]    12B index.html


################### 分割线 #################################

echo 'hello world' > hellWord.txt
mc cp ./hellWord.txt minio-cluster-2/c2-n2-bk1
mc ls minio-cluster-1/c1-n1-bk1


# 看到如下输出即代表c1-n1-bk1 可以同步数据到 c2-n2-bk1
[2021-08-23 17:54:17 CST]    12B hellWord.txt

至此,Game Over !!!

评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值