glusterfs 安装 【备忘】

说明

介绍 glusterfs 安装
不建议 glusterfs 用于大规模生产环境中
glusterfs 扩容过程中会打爆网络 （数据迁移，客户连接共享同一个网络通道）
官方文档会建议在扩容过程中需要断开所有客户端连接

机器说明

主机名	ip	角色
ns-storage-020103.vclound.com	1.1.1.3	glusterfs 服务器
ns-storage-020104.vclound.com	1.1.1.4	glusterfs 服务器
ns-storage-020105.vclound.com	1.1.1.5	glusterfs 服务器

部署 glusterfs

安装准备

主机需要 dns 解析
ntp 时间同步
关闭 selinux, firewalld 服务

软件源

https://download.gluster.org/pub/gluster/glusterfs/

软件包安装

# yum install centos-release-gluster
# yum install glusterfs-server

服务启动

服务启动后才可以通过命令行找寻到其他服务器并加入集群

 service glusterd start

添加其他服务器

# gluster peer probe ns-storage-020104.vclound.com
   peer probe: success
# gluster peer probe ns-storage-020105.vclound.com
   peer probe: success

ns-storage-020104, ns-storage-020105 执行

# gluster peer probe ns-storage-020103.vclound.com
   peer probe: success

健康检查

# gluster peer status
Number of Peers: 2

Hostname: ns-storage-020104.vclound.com
Uuid: 5f531cef-3de0-4869-824d-a7691f946598
State: Peer in Cluster (Connected)

Hostname: ns-storage-020105.vclound.com
Uuid: 9220b8b6-aaae-40e7-be6c-f60fed490d72
State: Peer in Cluster (Connected)

创建卷

注意需要独立磁盘挂载到 /data/brick1 目录下
分别在三个服务器上执行

mkdir -p /data/brick1/gv0

在任意服务器上执行

# gluster volume create gv0 replica 3 ns-storage-020103.vclound.com:/data/brick1/gv0 ns-storage-020104.vclound.com:/data/brick1/gv0 ns-storage-020105.vclound.com:/data/brick1/gv0
     volume create: gv0: success: please start the volume to access data

启动卷

# gluster volume status gv0
Volume gv0 is not started
# gluster volume start gv0
volume start: gv0: success

检查卷状态

# gluster volume status gv0
Status of volume: gv0
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick ns-storage-020103.vclound.com:/data/brick1/gv0 49152 0 Y 12778
Brick ns-storage-020104.vclound.com:/data/brick1/gv0 49152 0 Y 12052
Brick ns-storage-020105.vclound.com:/data/brick1/gv0 49152 0 Y 4592
Self-heal Daemon on localhost N/A N/A Y 12795
Self-heal Daemon on ns-storage-020104.vclound.com N/A N/A Y 12069
Self-heal Daemon on ns-storage-020105.vclound.com N/A N/A Y 4609

Task Status of Volume gv0
------------------------------------------------------------------------------
There are no active volume tasks

卷状态监测

gluster volume info gv0

Volume Name: gv0
Type: Replicate
Volume ID: 3ea3ca86-601c-4f10-9bd9-0b17fbf012f3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ns-storage-020103.vclound.com:/data/brick1/gv0
Brick2: ns-storage-020104.vclound.com:/data/brick1/gv0
Brick3: ns-storage-020105.vclound.com:/data/brick1/gv0
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

卷健康状态

# gluster volume heal gv0 info
Brick ns-storage-020103.vclound.com:/data/brick1/gv0
Status: Connected
Number of entries: 0

Brick ns-storage-020104.vclound.com:/data/brick1/gv0
Status: Connected
Number of entries: 0

Brick ns-storage-020105.vclound.com:/data/brick1/gv0
Status: Connected
Number of entries: 0

常见故障

卷断开连接

假如健康状态有问题，参考下面 example

# gluster volume heal gv0 info
Brick ns-storage-020103.vclound.com:/data/brick1/gv0
Status: Connected
Number of entries: 0

Brick ns-storage-020104.vclound.com:/data/brick1/gv0
Status: Connected
Number of entries: 0

Brick ns-storage-020105.vclound.com:/data/brick1/gv0
Status: 传输端点尚未连接
Number of entries: -

解决上述问题方法

# gluster volume stop gv0
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: gv0: success
# gluster volume start gv0
volume start: gv0: success
# gluster volume heal gv0 info
Brick ns-storage-020103.vclound.com:/data/brick1/gv0
Status: Connected
Number of entries: 0

Brick ns-storage-020104.vclound.com:/data/brick1/gv0
Status: Connected
Number of entries: 0

Brick ns-storage-020105.vclound.com:/data/brick1/gv0
Status: Connected
Number of entries: 0

节点故障

参考下面信息
正常装填下， Online 都应该显示为 Y

# gluster volume status gv0
Status of volume: gv0
Gluster process                                                                       TCP Port  RDMA Port  Online  Pid
---------------------------------------------------------------------------------------------------------------------------
Brick ns-storage-020103.vclound.com:/data/brick1/gv0                                   N/A       N/A        N       N/A
Brick ns-storage-020104.vclound.com:/data/brick1/gv0                                   N/A       N/A        N       N/A
Brick ns-storage-020105.vclound.com:/data/brick1/gv0                                   N/A       N/A        N       N/A
Self-heal Daemon on localhost                                                          N/A       N/A        Y       7967
Self-heal Daemon on ns-storage-020104.vclound.com                                      N/A       N/A        Y       56792
Self-heal Daemon on ns-storage-020105.vclound.com                                      N/A       N/A        Y       49413

Task Status of Volume gv0
---------------------------------------------------------------------------------------------------------------------------
There are no active volume tasks

解决方法

# gluster volume start gv0
volume start: gv0: failed: Volume gv0 already started
# gluster volume start gv0 force
volume start: gv0: success

# gluster volume status gv0
Status of volume: gv0
Gluster process                                                                       TCP Port  RDMA Port  Online  Pid
--------------------------------------------------------------------------------------------------------------------------
Brick ns-storage-020103.vclound.com:/data/brick1/gv0                                   49160     0          Y       29736
Brick ns-storage-020104.vclound.com:/data/brick1/gv0                                   49164     0          Y       76081
Brick ns-storage-020105.vclound.com:/data/brick1/gv0                                   49162     0          Y       85769
Self-heal Daemon on localhost                                                          N/A       N/A        Y       7967
Self-heal Daemon on ns-storage-020104.vclound.com                                      N/A       N/A        Y       56792
Self-heal Daemon on ns-storage-020105.vclound.com                                      N/A       N/A        Y       49413

Task Status of Volume gv0
--------------------------------------------------------------------------------------------------------------------------
There are no active volume tasks