glusterfs存储卷分布卷、复制卷测试

最新推荐文章于 2024-09-18 01:42:39 发布

daydayup9527

最新推荐文章于 2024-09-18 01:42:39 发布

阅读量463

点赞数

分类专栏： ceph存储，gluster存储文章标签：运维 linux 学习

本文链接：https://blog.csdn.net/weixin_60092693/article/details/123964849

版权

ceph存储，gluster存储专栏收录该内容

7 篇文章 0 订阅

订阅专栏

本文详细介绍了如何使用GlusterFS创建和管理分布式卷，包括创建分布卷和复制卷，挂载测试，添加和删除brick，数据一致性修复，以及在节点宕机情况下的自动修复。此外，还涵盖了卷的扩容、缩减、删除，目录大小限制以及主机故障恢复的模拟过程。

摘要由CSDN通过智能技术生成

分布式卷

创建分布卷默认模式类似raid0，hash写入单个文件，节点上是单个文件

[root@node1 ~]# gluster volume create test1 node1:/brick/test1
volume create: test1: success: please start the volume to access data
[root@node1 ~]# gluster volume info test1
Bricks:
Brick1: node1:/brick/test1
[root@node1 ~]# gluster volume start test1
volume start: test1: success

挂载测试

[root@client ~]# mount.glusterfs node1:/test1 /opt
[root@client ~]# touch /opt/test{1..5}
[root@client ~]# ls /opt/
test1  test2  test3  test4  test5
[root@node1 ~]# ls /brick/test1/
test1  test2  test3  test4  test5

增加brick

[root@node1 ~]# gluster volume add-brick test1 node2:/brick/test1
volume add-brick: success
[root@node1 ~]# gluster volume info test1
Bricks:
Brick1: node1:/brick/test1
Brick2: node2:/brick/test1

rebalance操作能够让文件按照之前的规则再分配现网中rebalance，最好在服务器空闲的时间操作

[root@node2 ~]# ls /brick/test1/      //此时是空的
[root@node1 ~]# gluster volume rebalance test1 fix-layout start
[root@node1 ~]# gluster volume rebalance test1 start
[root@node1 ~]# ls /brick/test1/
test1  test2  test4  test5
[root@node1 ~]# gluster volume rebalance test1 status
volume rebalance: test1: success
[root@node2 ~]# ls /brick/test1/     //此时有文件被分到node2节点上去了
test3
[root@node1 ~]# ls /brick/test1/
test1  test2  test4  test5

创建复制卷

创建卷

[root@node1 ~]# gluster volume create test_rep replica 2 node1:/brick/test-rep   node2:/brick/test-rep
[root@node1 ~]# gluster volume info test_rep
Bricks:
Brick1: node1:/brick/test-rep
Brick2: node2:/brick/test-rep
[root@node1 ~]# gluster volume start test_rep
volume start: test_rep: success

挂在卷

[root@client ~]# mkdir /test_rep

[root@client ~]# mount.glusterfs node1:/test_rep /test_rep/
[root@client ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
node1:/test_rep           20G  245M   19G   2% /test_rep

测试卷

[root@client ~]# touch /test_rep/test{1..5}
[root@client ~]# ls /test_rep/
test1  test2  test3  test4  test5
[root@node1 ~]# ls /brick/test-rep/
test1  test2  test3  test4  test5
[root@node2 ~]# ls /brick/test-rep/
test1  test2  test3  test4  test5

添加副本

[root@node1 ~]# gluster volume add-brick test_rep replica 3 node3:/brick/test-rep
volume add-brick: success
[root@node3 ~]# ls /brick/test-rep/
test1  test2  test3  test4  test5

删除副本

[root@node1 ~]# gluster volume remove-brick test_rep replica 2 node3:/brick/test-rep force
[root@node1 ~]# gluster volume info test_rep
Bricks:
Brick1: node1:/brick/test-rep
Brick2: node2:/brick/test-rep

替换副本

[root@node3 ~]# rm -rf /brick/test-rep/

[root@node1 ~]# gluster volume replace-brick test_rep node2:/brick/test-rep node3:/brick/test-rep commit force
[root@node1 ~]# gluster volume info test_rep
Bricks:
Brick1: node1:/brick/test-rep
Brick2: node3:/brick/test-rep
[root@node3 ~]# ls /brick/test-rep/
test1  test2  test3  test4  test5

模拟数据删除恢复

[root@node2 ~]# rm -rf /brick/test-rep/test1
[root@node2 ~]# ls /brick/test-rep/
test2  test3  test4  test5

[root@node1 ~]# gluster volume heal test_rep full
[root@node2 ~]# ls /brick/test-rep/
test1  test2  test3  test4  test5

[root@node1 ~]# gluster volume heal test_rep  info     //查看需要修复的文件
Brick node1:/brick/test-rep
Status: Connected
Number of entries: 0
Brick node2:/brick/test-rep
Status: Connected
Number of entries: 0
[root@node1 ~]# gluster volume heal test_rep  info split-brain   //脑裂文件
Brick node1:/brick/test-rep
Status: Connected
Number of entries in split-brain: 0

Brick node2:/brick/test-rep
Status: Connected
Number of entries in split-brain: 0

GlusterFS配置信息和日志

[root@node1 ~]# ls /var/lib/glusterd/       //配置信息
bitd           glusterd.upgrade  glustershd  hooks  options  quotad  snaps     vols
glusterd.info  glusterfind       groups      nfs    peers    scrub   ss_brick
[root@node1 ~]# ls /var/lib/glusterd/vols/        //创建出来的卷信息位置
test1  test_rep
[root@node1 ~]# ls /var/log/glusterfs/    //日志信息
bricks  cli.log  cmd_history.log  gfproxy  glusterd.log  glustershd.log  snaps  start.log  status.log  test1-rebalance.log

Gluster设置访问客户端IP

[root@node1 ~]# gluster volume set test_rep auth.allow 192.168.1.*
[root@node1 ~]# gluster volume info test_rep
Options Reconfigured:
auth.allow: 192.168.1.*

复制卷数据不一致修复处理

[root@client ~]# ls /opt/
test1  test2  test3  test4  test5
[root@node2 ~]# ls /brick/test-rep/
test1  test2  test3  test4  test5
[root@node2 ~]# rm -rf /brick/test-rep/test1
[root@node2 ~]# ls /brick/test-rep/
test2  test3  test4  test5

[root@client ~]# cat /opt/test1   //访问挂载目录里的文件,能正常访问

[root@node2 ~]# ls /brick/test-rep/ 
test1  test2  test3  test4  test5      //访问文件触发文件的自动修复

节点宕机自动修复

[root@node2 ~]# systemctl stop glusterd.service
[root@client ~]# touch /opt/first-test
[root@node2 ~]# ls /brick/test-rep/
test1  test2  test3  test4  test5
[root@node2 ~]# systemctl start  glusterd.service
[root@node2 ~]# ls /brick/test-rep/
first-test  test1  test2  test3  test4  test5

复制卷的扩容


[root@node1 ~]# gluster volume add-brick test-rep replica 2 node3:/brick/test-rep node4:/brick/test-rep  force   
#添加两个块设备
[root@node1 ~]# gluster volume info test-rep
Number of Bricks: 2 x 2 = 4         #已经扩容

缩减与删除

[root@node1 ~]# gluster volume stop test-rep    //先停止卷
#然后移除卷，replica为2，移除必须是2的倍数
[root@node1 ~]# gluster volume remove-brick test-rep replica 2 node3:/brick/test-rep node4:/brick/test-rep force			
[root@node1 ~]# gluster volume info test-rep			//检测
Status: Stopped
Number of Bricks: 1 x 2 = 2
[root@glusterfs01 ~]# gluster volume start test-rep     //重新启动卷

[root@node1 ~]# gluster volume stop test-rep
[root@node1 ~]# gluster volume delete   test-rep    //删除卷

限制目录大小

[root@node1 ~]# gluster volume quota test_rep  enable     //开启
volume quota : success
[root@node1 ~]# gluster volume quota test_rep  disable		//关闭

[root@client ~]# mkdir /opt/limit

[root@node1 ~]# gluster volume quota test_rep  limit-usage  /limit    10GB
volume quota : success
[root@node1 ~]# gluster volume quota  test_rep list
 Path      Hard-limit  Soft-limit      Used  Available  Soft-limit exceeded? Hard-limit 
-----------------------------------------------------------------------------------------
/limit     10.0GB     80%(8.0GB)   0Bytes  10.0GB              No                   No

[root@node1 ~]# gluster volume quota test_rep remove /limit    //移除使用大小限制
volume quota : success
[root@node1 ~]# gluster volume quota  test_rep list
quota: No quota configured on volume test_rep

主机故障恢复的模拟

找一台完全一样的机器，配置和故障机同样的ip，安装gluster软件，保证配置一样，
gluster peer status  //主节点上查看故障机器的Uuid
vim /var/lib/glusterd/glusterd.info   //修改新装的机器的UUID信息
gluster volume heal test_rep full     //启动后执行修复