linux部署minio对象存储集群

最新推荐文章于 2024-07-25 15:19:22 发布

willops

最新推荐文章于 2024-07-25 15:19:22 发布

阅读量4.1k

点赞数 2

分类专栏： storage 文章标签： minio

本文链接：https://blog.csdn.net/networken/article/details/111505113

版权

storage 专栏收录该内容

10 篇文章 5 订阅

订阅专栏

minio简介

MinIO 是一个基于Apache License v2.0开源协议的对象存储服务。它兼容亚马逊S3云存储服务接口，非常适合于存储大容量非结构化的数据，例如图片、视频、日志文件、备份数据和容器/虚拟机镜像等，而一个对象文件可以是任意大小，从几kb到最大5T不等。
在这里插入图片描述
官网：https://min.io/

minio启动模式：

standalone FS mode：不启用纠删码机制，minio server后只能直接指定disk path，并且disk path不能使用http://endpoint类型，只能挂载单个磁盘目录，示例：
```
minio server /mnt/data
```

standalone erasure mode：启用纠删码机制，单个节点启动多个磁盘，最小磁盘数为4，示例：

minio server /mnt/data{1...64}
mc admin info minio --json | jq .info.backend #查看是否启用纠删码机制

distributed mode：启用纠删码机制，分布式集群模式，总最小磁盘数量为4，最小节点数为2，示例：
```
minio server http://host{1...n}/export{1...m}
```

erasure Code模式单租户下minio对节点及磁盘要求：

单个集群最大节点数无限制(minio已取消32节点限制)
单个节点最大磁盘数量无限制
集群数据不丢失需要最小节点数为2
仅有1个节点时需要最小磁盘数为4
2个或3个节点时每个节点最小磁盘数为2
4个节点时每个节点最小磁盘数为1
保证集群可读的最小在线磁盘数为N/2，N为所有节点总磁盘数
保证集群可写的最小在线磁盘数为N/2+1，N为所有节点总磁盘数

最小集群模式故障场景：

2个节点故障1个节点时，集群可读不可写
3个节点故障1个节点时，集群可读可写，故障2个节点时集群不可用
4个节点故障1个节点时，集群可读可写，故障2个节点时集群可读不可写

minio集群部署

官方推荐部署distributed模式时最少节点数为4个：

192.168.93.40 minio-1
192.168.93.41 minio-2
192.168.93.42 minio-3
192.168.93.43 minio-4

实际部署需要准备磁盘，以每个节点挂载一块/dev/sdb盘为例，格式化磁盘并挂载至/data目录下，这里作为演示直接使用系统盘根目录，省略以下步骤。

lsblk
mkdir /data
mkfs.xfs /dev/sdb
echo "/dev/sdb /data xfs defaults,noatime,nofail 0 0" >> /etc/fstab
mount -a
df -h

以下操作在所有节点执行。

配置主机名

hostnamectl set-hostname minio-xx

配置/etc/hosts

cat >> /etc/hosts << EOF
192.168.93.40 minio-1
192.168.93.41 minio-2
192.168.93.42 minio-3
192.168.93.43 minio-4
EOF

关闭firewall防火墙

systemctl disable --now firewalld

关闭selinux

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config

配置时间同步

yum install -y chrony
systemctl enable --now chronyd
timedatectl

安装minio

wget -O /usr/local/bin/minio https://dl.minio.io/server/minio/release/linux-amd64/minio
chmod +x /usr/local/bin/minio

创建用户和目录

useradd -r minio-user -s /sbin/nologin
mkdir /data
chown minio-user:minio-user /data

创建minio服务启动参数

参考：https://github.com/minio/minio-service/blob/master/linux-systemd/README.md

cat > /etc/default/minio << EOF
MINIO_OPTS="--address :9000 http://minio-1/data http://minio-2/data http://minio-3/data http://minio-4/data"
MINIO_ACCESS_KEY=minio
MINIO_SECRET_KEY=minio123
EOF

下载官方systemd启动文件

( cd /etc/systemd/system/; curl -O https://raw.githubusercontent.com/minio/minio-service/master/linux-systemd/minio.service )

修改为以下内容：

cat > /etc/systemd/system/minio.service <<'EOF'
[Unit]
Description=MinIO
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local/

User=minio-user
Group=minio-user

EnvironmentFile=/etc/default/minio
ExecStart=/usr/local/bin/minio server $MINIO_OPTS

Restart=always
LimitNOFILE=65536
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target
EOF

启动minio服务并配置为开机启动

systemctl enable --now minio

查看minio服务运行状态，其中 Status: 4 Online, 0 Offline.说明所有节点全部在线：

[root@minio-1 ~]# systemctl status minio
● minio.service - MinIO
   Loaded: loaded (/etc/systemd/system/minio.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-12-22 15:07:37 CST; 13s ago
     Docs: https://docs.min.io
 Main PID: 7946 (minio)
   CGroup: /system.slice/minio.service
           └─7946 /usr/local/bin/minio server --address :9000 http://minio-1/data http://minio-2/data http://minio-3/data http://minio-4/data

Dec 22 15:07:40 minio-1 minio[7946]: Status:         4 Online, 0 Offline.
Dec 22 15:07:40 minio-1 minio[7946]: Endpoint:  http://192.168.93.40:9000  http://127.0.0.1:9000
Dec 22 15:07:40 minio-1 minio[7946]: Browser Access:
Dec 22 15:07:40 minio-1 minio[7946]: http://192.168.93.40:9000  http://127.0.0.1:9000
Dec 22 15:07:40 minio-1 minio[7946]: Object API (Amazon S3 compatible):
Dec 22 15:07:40 minio-1 minio[7946]: Go:         https://docs.min.io/docs/golang-client-quickstart-guide
Dec 22 15:07:40 minio-1 minio[7946]: Java:       https://docs.min.io/docs/java-client-quickstart-guide
Dec 22 15:07:40 minio-1 minio[7946]: Python:     https://docs.min.io/docs/python-client-quickstart-guide
Dec 22 15:07:40 minio-1 minio[7946]: JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
Dec 22 15:07:40 minio-1 minio[7946]: .NET:       https://docs.min.io/docs/dotnet-client-quickstart-guide

配置负载均衡

使用nginx和keepalived实现负载均衡。实际部署需要准备2台服务器作为负载均衡节点，这里作为演示将minio-1和minio-2节点复用为负载均衡节点。

在2个节点安装nginx和keepalived：

yum install -y nginx keepalived

创建nginx配置文件，修改server地址并自定义listen端口，注意修改server_name参数为对应节点IP：

cat > /etc/nginx/conf.d/minio-lb.conf <<'EOF'
upstream minio_server {
    server 192.168.93.40:9000;
    server 192.168.93.41:9000;
    server 192.168.93.42:9000;
    server 192.168.93.43:9000;
    }

server {
    listen 9001;
    server_name  192.168.93.40;
    ignore_invalid_headers off;
    client_max_body_size 0;
    proxy_buffering off;
    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_connect_timeout 300;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;
        proxy_pass http://minio_server;
    }
}
EOF

启动nginx服务

systemctl enable --now nginx

创建keepalived配置文件，根据实际环境修改interface及virtual_ipaddress参数，2个节点配置相同:

cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived

global_defs {
   router_id minio
   vrrp_version 2
   vrrp_garp_master_delay 1
   script_user root
   enable_script_security 
}

vrrp_script chk_nginx {
    script       "pidof nginx"
    timeout 3
    interval 3   # check every 1 second
    fall 2       # require 2 failures for KO
    rise 2       # require 2 successes for OK
}

vrrp_instance lb-minio {
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.93.50
    }
    track_script {
        chk_nginx
    }
}
EOF

启动keepalvied服务

systemctl enable --now keepalived

查看创建的vip

[root@minio-1 ~]# ip a |grep 192.168.93
    inet 192.168.93.40/24 brd 192.168.93.255 scope global noprefixroute ens33
    inet 192.168.93.50/32 scope global ens33

关闭vip所在节点nginx模拟负载均衡故障:

systemctl stop nginx

查看keepalived日志

[root@minio-1 ~]# journalctl -xeu keepalived
......
Dec 22 15:23:53 minio-1 Keepalived_vrrp[2368]: VRRP_Script(chk_nginx) failed
Dec 22 15:23:53 minio-1 Keepalived_vrrp[2368]: VRRP_Instance(lb-minio) Entering FAULT STATE
Dec 22 15:23:53 minio-1 Keepalived_vrrp[2368]: VRRP_Instance(lb-minio) removing protocol VIPs.
Dec 22 15:23:53 minio-1 Keepalived_vrrp[2368]: VRRP_Instance(lb-minio) Now in FAULT state
Dec 22 15:23:56 minio-1 Keepalived_vrrp[2368]: /usr/sbin/pidof nginx exited with status 1

查看vip已经迁移至另一个节点

[root@minio-2 ~]# ip a |grep 192.168.93.
    inet 192.168.93.41/24 brd 192.168.93.255 scope global noprefixroute ens33
    inet 192.168.93.50/32 scope global ens33

使用vip访问minio UI
在这里插入图片描述

mc mirror配置

mc mirror -w参数可实时同步本地数据到minio对象存储，保证本地文件的所有增删改查都会与minio目标端同步。

下载mc客户端

wget -O /usr/local/bin/mc https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x /usr/local/bin/mc

实测最新版本mc客户端可能存在bug，无法使用mc mirror -w参数，以下版本经测试可用：

https://dl.min.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2020-11-17T00-39-14Z

配置认证信息

mc config host add minio http://192.168.93.40:9000 minio minio123

创建bucket

mc mb minio/nfsbak

实时同步/mnt/nfs目录下内容到minio bucket中

mc mirror -w --overwrite --remove /mnt/nfs minio/nfsbak

配置使用systemd管理服务，注意修改Environment参数：

cat > /etc/systemd/system/minioc.service <<'EOF'
[Unit]
Description=minioc
After=network.target

[Service]
Type=simple
Environment=backup="/mnt/nfs"
Environment=bucket="nfsbak"
ExecStart=/bin/bash -c "mc mirror -w --overwrite --remove ${backup} minio/${bucket}"
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动服务并设为开机随系统启动

systemctl enable --now minioc

查看服务运行状态

[root@master ~]# systemctl status minioc    
● minioc.service - minioc
   Loaded: loaded (/etc/systemd/system/minioc.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-12-28 15:11:08 CST; 3s ago
 Main PID: 3621799 (mc)
    Tasks: 8 (limit: 49791)
   Memory: 12.0M
   CGroup: /system.slice/minioc.service
           └─3621799 mc mirror -w --overwrite --remove /mnt/nfs minio/nfsbak

Dec 28 15:11:08 master systemd[1]: Started minioc.
Dec 28 15:11:08 master bash[3621799]: Removing `minio/nfsbak/789`.
Dec 28 15:11:08 master bash[3621799]: Removing `minio/nfsbak/456`.

在/mnt/nfs目录下写入或删除数据，查看日志，文件会自动同步到minio

[root@master ~]# journalctl -xeu minioc
......
-- The start-up result is done.
Dec 25 17:00:57 master mc[553138]: `/mnt/nfs/123` -> `minio/nfsbak/123`
Dec 25 17:01:09 master mc[553138]: `/mnt/nfs/789` -> `minio/nfsbak/789`
Dec 25 17:02:04 master mc[553138]: Removing `minio/nfsbak/123`.
lines 449-487/487 (END)

rclone对接minio

安装rclone

curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
cp rclone-*-linux-amd64/rclone /usr/bin/

创建rclone配置文件

[root@minio-1 ~]# cat /root/.config/rclone/rclone.conf 
[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = minio
secret_access_key = minio123
endpoint = http://192.168.93.40:9000

列出bucket

rclone lsd minio:

新建bucket

rclone mkdir minio:bucket

复制文件到bucket

rclone copy /root/test.txt minio:bucket

复制bucket文件到本地

rclone copy minio:bucket /tmp/bucket-copy

列出bucket中所有文件

rclone ls minio:bucket

同步文件到bucket

rclone sync /root/test.txt minio:bucket

挂载minio到本地

yum install -y fuse
mkdir -p /mnt/minio

#挂载整个bucket
rclone mount minio:bucket /mnt/minio

#后台运行
rclone mount --daemon minio:bucket/test/ /mnt/minio/

#解除挂载
fusermount -u /path/to/local/mount

willops

关注

2
点赞
踩
14

收藏

觉得还不错? 一键收藏
6
评论
linux部署minio对象存储集群

minio简介MinIO 是一个基于Apache License v2.0开源协议的对象存储服务。它兼容亚马逊S3云存储服务接口，非常适合于存储大容量非结构化的数据，例如图片、视频、日志文件、备份数据和容器/虚拟机镜像等，而一个对象文件可以是任意大小，从几kb到最大5T不等。官网：https://min.io/集群部署准备4个节点192.168.93.40 minio-1192.168.93.41 minio-2192.168.93.42 minio-3192.168.93.43 mini
复制链接

扫一扫

专栏目录