minio简介
MinIO 是一个基于Apache License v2.0开源协议的对象存储服务。它兼容亚马逊S3云存储服务接口,非常适合于存储大容量非结构化的数据,例如图片、视频、日志文件、备份数据和容器/虚拟机镜像等,而一个对象文件可以是任意大小,从几kb到最大5T不等。
官网:https://min.io/
minio启动模式:
-
standalone FS mode:不启用纠删码机制,minio server后只能直接指定disk path,并且disk path不能使用http://endpoint类型,只能挂载单个磁盘目录,示例:
minio server /mnt/data
-
standalone erasure mode:启用纠删码机制,单个节点启动多个磁盘,最小磁盘数为4,示例:
minio server /mnt/data{1...64} mc admin info minio --json | jq .info.backend #查看是否启用纠删码机制
-
distributed mode:启用纠删码机制,分布式集群模式,总最小磁盘数量为4,最小节点数为2,示例:
minio server http://host{1...n}/export{1...m}
erasure Code模式单租户下minio对节点及磁盘要求:
- 单个集群最大节点数无限制(minio已取消32节点限制)
- 单个节点最大磁盘数量无限制
- 集群数据不丢失需要最小节点数为2
- 仅有1个节点时需要最小磁盘数为4
- 2个或3个节点时每个节点最小磁盘数为2
- 4个节点时每个节点最小磁盘数为1
- 保证集群可读的最小在线磁盘数为N/2,N为所有节点总磁盘数
- 保证集群可写的最小在线磁盘数为N/2+1,N为所有节点总磁盘数
最小集群模式故障场景:
- 2个节点故障1个节点时,集群可读不可写
- 3个节点故障1个节点时,集群可读可写,故障2个节点时集群不可用
- 4个节点故障1个节点时,集群可读可写,故障2个节点时集群可读不可写
minio集群部署
官方推荐部署distributed模式时最少节点数为4个:
192.168.93.40 minio-1
192.168.93.41 minio-2
192.168.93.42 minio-3
192.168.93.43 minio-4
实际部署需要准备磁盘,以每个节点挂载一块/dev/sdb盘为例,格式化磁盘并挂载至/data目录下,这里作为演示直接使用系统盘根目录,省略以下步骤。
lsblk
mkdir /data
mkfs.xfs /dev/sdb
echo "/dev/sdb /data xfs defaults,noatime,nofail 0 0" >> /etc/fstab
mount -a
df -h
以下操作在所有节点执行。
配置主机名
hostnamectl set-hostname minio-xx
配置/etc/hosts
cat >> /etc/hosts << EOF
192.168.93.40 minio-1
192.168.93.41 minio-2
192.168.93.42 minio-3
192.168.93.43 minio-4
EOF
关闭firewall防火墙
systemctl disable --now firewalld
关闭selinux
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config
配置时间同步
yum install -y chrony
systemctl enable --now chronyd
timedatectl
安装minio
wget -O /usr/local/bin/minio https://dl.minio.io/server/minio/release/linux-amd64/minio
chmod +x /usr/local/bin/minio
创建用户和目录
useradd -r minio-user -s /sbin/nologin
mkdir /data
chown minio-user:minio-user /data
创建minio服务启动参数
参考:https://github.com/minio/minio-service/blob/master/linux-systemd/README.md
cat > /etc/default/minio << EOF
MINIO_OPTS="--address :9000 http://minio-1/data http://minio-2/data http://minio-3/data http://minio-4/data"
MINIO_ACCESS_KEY=minio
MINIO_SECRET_KEY=minio123
EOF
下载官方systemd启动文件
( cd /etc/systemd/system/; curl -O https://raw.githubusercontent.com/minio/minio-service/master/linux-systemd/minio.service )
修改为以下内容:
cat > /etc/systemd/system/minio.service <<'EOF'
[Unit]
Description=MinIO
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio
[Service]
WorkingDirectory=/usr/local/
User=minio-user
Group=minio-user
EnvironmentFile=/etc/default/minio
ExecStart=/usr/local/bin/minio server $MINIO_OPTS
Restart=always
LimitNOFILE=65536
TimeoutStopSec=infinity
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
EOF
启动minio服务并配置为开机启动
systemctl enable --now minio
查看minio服务运行状态,其中 Status: 4 Online, 0 Offline.说明所有节点全部在线:
[root@minio-1 ~]# systemctl status minio
● minio.service - MinIO
Loaded: loaded (/etc/systemd/system/minio.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2020-12-22 15:07:37 CST; 13s ago
Docs: https://docs.min.io
Main PID: 7946 (minio)
CGroup: /system.slice/minio.service
└─7946 /usr/local/bin/minio server --address :9000 http://minio-1/data http://minio-2/data http://minio-3/data http://minio-4/data
Dec 22 15:07:40 minio-1 minio[7946]: Status: 4 Online, 0 Offline.
Dec 22 15:07:40 minio-1 minio[7946]: Endpoint: http://192.168.93.40:9000 http://127.0.0.1:9000
Dec 22 15:07:40 minio-1 minio[7946]: Browser Access:
Dec 22 15:07:40 minio-1 minio[7946]: http://192.168.93.40:9000 http://127.0.0.1:9000
Dec 22 15:07:40 minio-1 minio[7946]: Object API (Amazon S3 compatible):
Dec 22 15:07:40 minio-1 minio[7946]: Go: https://docs.min.io/docs/golang-client-quickstart-guide
Dec 22 15:07:40 minio-1 minio[7946]: Java: https://docs.min.io/docs/java-client-quickstart-guide
Dec 22 15:07:40 minio-1 minio[7946]: Python: https://docs.min.io/docs/python-client-quickstart-guide
Dec 22 15:07:40 minio-1 minio[7946]: JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
Dec 22 15:07:40 minio-1 minio[7946]: .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
配置负载均衡
使用nginx和keepalived实现负载均衡。实际部署需要准备2台服务器作为负载均衡节点,这里作为演示将minio-1和minio-2节点复用为负载均衡节点。
在2个节点安装nginx和keepalived:
yum install -y nginx keepalived
创建nginx配置文件,修改server地址并自定义listen端口,注意修改server_name参数为对应节点IP:
cat > /etc/nginx/conf.d/minio-lb.conf <<'EOF'
upstream minio_server {
server 192.168.93.40:9000;
server 192.168.93.41:9000;
server 192.168.93.42:9000;
server 192.168.93.43:9000;
}
server {
listen 9001;
server_name 192.168.93.40;
ignore_invalid_headers off;
client_max_body_size 0;
proxy_buffering off;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_connect_timeout 300;
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio_server;
}
}
EOF
启动nginx服务
systemctl enable --now nginx
创建keepalived配置文件,根据实际环境修改interface及virtual_ipaddress参数,2个节点配置相同:
cat > /etc/keepalived/keepalived.conf <<EOF
! Configuration File for keepalived
global_defs {
router_id minio
vrrp_version 2
vrrp_garp_master_delay 1
script_user root
enable_script_security
}
vrrp_script chk_nginx {
script "pidof nginx"
timeout 3
interval 3 # check every 1 second
fall 2 # require 2 failures for KO
rise 2 # require 2 successes for OK
}
vrrp_instance lb-minio {
state BACKUP
interface ens33
virtual_router_id 51
priority 100
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.93.50
}
track_script {
chk_nginx
}
}
EOF
启动keepalvied服务
systemctl enable --now keepalived
查看创建的vip
[root@minio-1 ~]# ip a |grep 192.168.93
inet 192.168.93.40/24 brd 192.168.93.255 scope global noprefixroute ens33
inet 192.168.93.50/32 scope global ens33
关闭vip所在节点nginx模拟负载均衡故障:
systemctl stop nginx
查看keepalived日志
[root@minio-1 ~]# journalctl -xeu keepalived
......
Dec 22 15:23:53 minio-1 Keepalived_vrrp[2368]: VRRP_Script(chk_nginx) failed
Dec 22 15:23:53 minio-1 Keepalived_vrrp[2368]: VRRP_Instance(lb-minio) Entering FAULT STATE
Dec 22 15:23:53 minio-1 Keepalived_vrrp[2368]: VRRP_Instance(lb-minio) removing protocol VIPs.
Dec 22 15:23:53 minio-1 Keepalived_vrrp[2368]: VRRP_Instance(lb-minio) Now in FAULT state
Dec 22 15:23:56 minio-1 Keepalived_vrrp[2368]: /usr/sbin/pidof nginx exited with status 1
查看vip已经迁移至另一个节点
[root@minio-2 ~]# ip a |grep 192.168.93.
inet 192.168.93.41/24 brd 192.168.93.255 scope global noprefixroute ens33
inet 192.168.93.50/32 scope global ens33
使用vip访问minio UI
mc mirror配置
mc mirror -w参数可实时同步本地数据到minio对象存储,保证本地文件的所有增删改查都会与minio目标端同步。
下载mc客户端
wget -O /usr/local/bin/mc https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x /usr/local/bin/mc
实测最新版本mc客户端可能存在bug,无法使用mc mirror -w参数,以下版本经测试可用:
https://dl.min.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2020-11-17T00-39-14Z
配置认证信息
mc config host add minio http://192.168.93.40:9000 minio minio123
创建bucket
mc mb minio/nfsbak
实时同步/mnt/nfs目录下内容到minio bucket中
mc mirror -w --overwrite --remove /mnt/nfs minio/nfsbak
配置使用systemd管理服务,注意修改Environment参数:
cat > /etc/systemd/system/minioc.service <<'EOF'
[Unit]
Description=minioc
After=network.target
[Service]
Type=simple
Environment=backup="/mnt/nfs"
Environment=bucket="nfsbak"
ExecStart=/bin/bash -c "mc mirror -w --overwrite --remove ${backup} minio/${bucket}"
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
启动服务并设为开机随系统启动
systemctl enable --now minioc
查看服务运行状态
[root@master ~]# systemctl status minioc
● minioc.service - minioc
Loaded: loaded (/etc/systemd/system/minioc.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-12-28 15:11:08 CST; 3s ago
Main PID: 3621799 (mc)
Tasks: 8 (limit: 49791)
Memory: 12.0M
CGroup: /system.slice/minioc.service
└─3621799 mc mirror -w --overwrite --remove /mnt/nfs minio/nfsbak
Dec 28 15:11:08 master systemd[1]: Started minioc.
Dec 28 15:11:08 master bash[3621799]: Removing `minio/nfsbak/789`.
Dec 28 15:11:08 master bash[3621799]: Removing `minio/nfsbak/456`.
在/mnt/nfs目录下写入或删除数据,查看日志,文件会自动同步到minio
[root@master ~]# journalctl -xeu minioc
......
-- The start-up result is done.
Dec 25 17:00:57 master mc[553138]: `/mnt/nfs/123` -> `minio/nfsbak/123`
Dec 25 17:01:09 master mc[553138]: `/mnt/nfs/789` -> `minio/nfsbak/789`
Dec 25 17:02:04 master mc[553138]: Removing `minio/nfsbak/123`.
lines 449-487/487 (END)
rclone对接minio
安装rclone
curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
cp rclone-*-linux-amd64/rclone /usr/bin/
创建rclone配置文件
[root@minio-1 ~]# cat /root/.config/rclone/rclone.conf
[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = minio
secret_access_key = minio123
endpoint = http://192.168.93.40:9000
列出bucket
rclone lsd minio:
新建bucket
rclone mkdir minio:bucket
复制文件到bucket
rclone copy /root/test.txt minio:bucket
复制bucket文件到本地
rclone copy minio:bucket /tmp/bucket-copy
列出bucket中所有文件
rclone ls minio:bucket
同步文件到bucket
rclone sync /root/test.txt minio:bucket
挂载minio到本地
yum install -y fuse
mkdir -p /mnt/minio
#挂载整个bucket
rclone mount minio:bucket /mnt/minio
#后台运行
rclone mount --daemon minio:bucket/test/ /mnt/minio/
#解除挂载
fusermount -u /path/to/local/mount