MFS分布式存储系统(3)-iscsi共享存储、master高可用部署、添加fence

最新推荐文章于 2024-03-14 09:18:35 发布

HTDST

最新推荐文章于 2024-03-14 09:18:35 发布

阅读量397

点赞数

分类专栏： Linux企业实战篇

本文链接：https://blog.csdn.net/nk298120/article/details/115954197

版权

MFS Master iscsi 高可用性 pacemaker fence机制

关键词由CSDN通过智能技术生成

Linux企业实战篇专栏收录该内容

87 篇文章 8 订阅

订阅专栏

文章目录

前言
一、环境准备
二、iscsi部署
- 服务端部署
- 客户端部署
三、两个master数据同步
四、结合pacemaker部署高可用
五、启用fence机制
- 服务端配置
- 客户端配置

前言

我们的目标是部署mfs的master高可用，首先我们需要搭建一个iscsi共享存储，然后同步两个master端的存储，之后将mfs master中的数据迁移到iscsi共享存储中，使得两个mfs master端的数据保持一致。

一、环境准备

server4上的chunk server停止，重新配置使其称为一个新的mfs master:
在这里插入图片描述
server3上新添加的硬盘分区删除掉，将mfs存储直接挂载到根分区上，使用空出来的硬盘做下一步的iscsi共享存储：

卸载之前挂载的硬盘，重启服务，mfs就挂载到了根分区上：

二、iscsi部署

服务端部署

我们选择server3作为服务端来配置iscsi共享存储

[root@server3 ~]# dd if=/dev/zero of=/dev/vdb bs=512 count=1     %清除磁盘上的分区表，因为iscsi输出的是存储块（具体什么原因还不理解）
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.000447115 s, 1.1 MB/s

[root@server3 ~]# yum install -y targetcli           %安装相关包
[root@server3 ~]# systemctl start target.service     %启动服务
targetcli shell version 2.1.fb46
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/backstores/block> ls
o- block ............................................................. [Storage Objects: 0]
/backstores/block> create my_disk /dev/vdb
Created block storage object my_disk using /dev/vdb.
/backstores/block> ls
o- block ............................................................. [Storage Objects: 1]
  o- my_disk .................................. [/dev/vdb (10.0GiB) write-thru deactivated]
    o- alua .............................................................. [ALUA Groups: 1]
      o- default_tg_pt_gp .................................. [ALUA state: Active/optimized]
/backstores/block> cd /
/> ls
o- / ................................................................................ [...]
  o- backstores ..................................................................... [...]
  | o- block ......................................................... [Storage Objects: 1]
  | | o- my_disk .............................. [/dev/vdb (10.0GiB) write-thru deactivated]
  | |   o- alua .......................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp .............................. [ALUA state: Active/optimized]
  | o- fileio ........................................................ [Storage Objects: 0]
  | o- pscsi ......................................................... [Storage Objects: 0]
  | o- ramdisk ....................................................... [Storage Objects: 0]
  o- iscsi ................................................................... [Targets: 0]
  o- loopback ................................................................ [Targets: 0]
/> cd /iscsi 
/iscsi> ls
o- iscsi ..................................................................... [Targets: 0]
/iscsi> 
/          @last      bookmarks  cd         create     delete     exit       get        
help       info       ls         pwd        refresh    set        status     version    
/iscsi> create iqn.2021-04.org.westos:storage1
Created target iqn.2021-04.org.westos:storage1.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/iscsi> ls
o- iscsi ..................................................................... [Targets: 1]
  o- iqn.2021-04.org.westos:storage1 ............................................ [TPGs: 1]
    o- tpg1 ........................................................ [no-gen-acls, no-auth]
      o- acls ................................................................... [ACLs: 0]
      o- luns ................................................................... [LUNs: 0]
      o- portals ............................................................. [Portals: 1]
        o- 0.0.0.0:3260 .............................................................. [OK]
/iscsi> cd iqn.2021-04.org.westos:storage1/
/iscsi/iqn.20...stos:storage1> ls
o- iqn.2021-04.org.westos:storage1 .............................................. [TPGs: 1]
  o- tpg1 .......................................................... [no-gen-acls, no-auth]
    o- acls ..................................................................... [ACLs: 0]
    o- luns ..................................................................... [LUNs: 0]
    o- portals ............................................................... [Portals: 1]
      o- 0.0.0.0:3260 ................................................................ [OK]
/iscsi/iqn.20...stos:storage1> cd tpg1/
/iscsi/iqn.20...storage1/tpg1> cd acls 
/iscsi/iqn.20...ge1/tpg1/acls> ls
o- acls ......................................................................... [ACLs: 0]
/iscsi/iqn.20...ge1/tpg1/acls> create iqn.2021-04.org.westos:client
Created Node ACL for iqn.2021-04.org.westos:client
/iscsi/iqn.20...ge1/tpg1/acls> ls
o- acls ......................................................................... [ACLs: 1]
  o- iqn.2021-04.org.westos:client ....................................... [Mapped LUNs: 0]
/iscsi/iqn.20...ge1/tpg1/acls> cd ..
/iscsi/iqn.20...storage1/tpg1> cd luns 
/iscsi/iqn.20...ge1/tpg1/luns> /backstores/block/my_disk/
/backstores/block/my_disk> ls
o- my_disk .................................... [/dev/vdb (10.0GiB) write-thru deactivated]
  o- alua ................................................................ [ALUA Groups: 1]
    o- default_tg_pt_gp .................................... [ALUA state: Active/optimized]
/backstores/block/my_disk> cd


/backstores/block/my_disk> ls
o- my_disk .................................... [/dev/vdb (10.0GiB) write-thru deactivated]
  o- alua ................................................................ [ALUA Groups: 1]
    o- default_tg_pt_gp .................................... [ALUA state: Active/optimized]
/backstores/block/my_disk> cd ..
/backstores/block> cd ..
/backstores> cd ..
/> ls
o- / ................................................................................ [...]
  o- backstores ..................................................................... [...]
  | o- block ......................................................... [Storage Objects: 1]
  | | o- my_disk .............................. [/dev/vdb (10.0GiB) write-thru deactivated]
  | |   o- alua .......................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp .............................. [ALUA state: Active/optimized]
  | o- fileio ........................................................ [Storage Objects: 0]
  | o- pscsi ......................................................... [Storage Objects: 0]
  | o- ramdisk ....................................................... [Storage Objects: 0]
  o- iscsi ................................................................... [Targets: 1]
  | o- iqn.2021-04.org.westos:storage1 .......................................... [TPGs: 1]
  |   o- tpg1 ...................................................... [no-gen-acls, no-auth]
  |     o- acls ................................................................. [ACLs: 1]
  |     | o- iqn.2021-04.org.westos:client ............................... [Mapped LUNs: 0]
  |     o- luns ................................................................. [LUNs: 0]
  |     o- portals ........................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ............................................................ [OK]
  o- loopback ................................................................ [Targets: 0]
/> cd /iscsi/iqn.2021-04.org.westos:storage1/
/iscsi/iqn.20...stos:storage1> ls
o- iqn.2021-04.org.westos:storage1 .............................................. [TPGs: 1]
  o- tpg1 .......................................................... [no-gen-acls, no-auth]
    o- acls ..................................................................... [ACLs: 1]
    | o- iqn.2021-04.org.westos:client ................................... [Mapped LUNs: 0]
    o- luns ..................................................................... [LUNs: 0]
    o- portals ............................................................... [Portals: 1]
      o- 0.0.0.0:3260 ................................................................ [OK]
/iscsi/iqn.20...stos:storage1> cd ..
/iscsi> cd iqn.2021-04.org.westos:storage1/tpg1/luns 
/iscsi/iqn.20...ge1/tpg1/luns> ls
o- luns ......................................................................... [LUNs: 0]
/iscsi/iqn.20...ge1/tpg1/luns> create /backstores/block/my_disk 
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.2021-04.org.westos:client
/iscsi/iqn.20...ge1/tpg1/luns> ls
o- luns ......................................................................... [LUNs: 1]
  o- lun0 ................................... [block/my_disk (/dev/vdb) (default_tg_pt_gp)]
/iscsi/iqn.20...ge1/tpg1/luns> cd ..
/iscsi/iqn.20...storage1/tpg1> cd ..
/iscsi/iqn.20...stos:storage1> ls
o- iqn.2021-04.org.westos:storage1 .............................................. [TPGs: 1]
  o- tpg1 .......................................................... [no-gen-acls, no-auth]
    o- acls ..................................................................... [ACLs: 1]
    | o- iqn.2021-04.org.westos:client ................................... [Mapped LUNs: 1]
    |   o- mapped_lun0 .......................................... [lun0 block/my_disk (rw)]
    o- luns ..................................................................... [LUNs: 1]
    | o- lun0 ............................... [block/my_disk (/dev/vdb) (default_tg_pt_gp)]
    o- portals ............................................................... [Portals: 1]
      o- 0.0.0.0:3260 ................................................................ [OK]
/iscsi/iqn.20...stos:storage1> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup/.
Configuration saved to /etc/target/saveconfig.json

在这里插入图片描述

客户端部署

在这里插入图片描述

server4上做同样的配置：

注意：iscsi属于本地文件系统，因此不支持多点同时挂载写入数据，因此一个节点挂载使用完之后一定要记得卸载

三、两个master数据同步

[root@server1 ~]# cd /var/lib/mfs/
[root@server1 mfs]# ls
changelog.1.mfs  changelog.4.mfs  metadata.crc         metadata.mfs.empty
changelog.2.mfs  changelog.5.mfs  metadata.mfs.back    stats.mfs
changelog.3.mfs  changelog.6.mfs  metadata.mfs.back.1
[root@server1 mfs]# cp -p * /mnt/
[root@server1 ~]# mount /dev/sda1 /mnt
[root@server1 ~]# cd /mnt/
[root@server1 mnt]# ls
changelog.1.mfs  changelog.4.mfs  metadata.crc         metadata.mfs.empty
changelog.2.mfs  changelog.5.mfs  metadata.mfs.back    stats.mfs
changelog.3.mfs  changelog.6.mfs  metadata.mfs.back.1
[root@server1 ~]# chown mfs.mfs /mnt
[root@server1 ~]# umount /mnt
[root@server1 ~]# systemctl stop moosefs-master.service 
[root@server1 ~]# cd /var/lib/mfs/
[root@server1 mfs]# ls                        %查看目录中的文件
changelog.1.mfs  changelog.4.mfs  metadata.crc         metadata.mfs.empty
changelog.2.mfs  changelog.5.mfs  metadata.mfs         stats.mfs
changelog.3.mfs  changelog.6.mfs  metadata.mfs.back.1
[root@server1 mnt]# cd            %做到这一步实际上mfs系统文件并没有同步过来，因为我们应该先关闭服务，再拷贝文件到目录中再挂载这个目录，因为服务关闭时会生成一个不同的文件metadata.mfs（原始镜像文件）     
[root@server1 ~]# mount /dev/sda1 /var/lib/mfs/     %挂载mfs的数据目录到共享存储上
[root@server1 ~]# df
Filesystem            1K-blocks    Used Available Use% Mounted on
/dev/mapper/rhel-root  17811456 1176828  16634628   7% /
devtmpfs                1011520       0   1011520   0% /dev
tmpfs                   1023544       0   1023544   0% /dev/shm
tmpfs                   1023544   17000   1006544   2% /run
tmpfs                   1023544       0   1023544   0% /sys/fs/cgroup
/dev/vda1               1038336  135088    903248  14% /boot
tmpfs                    204712       0    204712   0% /run/user/0
/dev/sda1              10474496   37908  10436588   1% /var/lib/mfs
[root@server1 ~]# cd /var/lib/mfs/
[root@server1 mfs]# ls                      %查看目录中的文件
changelog.1.mfs  changelog.4.mfs  metadata.crc         metadata.mfs.empty
changelog.2.mfs  changelog.5.mfs  metadata.mfs.back    stats.mfs
changelog.3.mfs  changelog.6.mfs  metadata.mfs.back.1
[root@server1 mfs]# systemctl start moosefs-master
Job for moosefs-master.service failed because the control process exited with error code. See "systemctl status moosefs-master.service" and "journalctl -xe" for details.
[root@server1 mfs]# ls
changelog.1.mfs  changelog.4.mfs  metadata.crc         metadata.mfs.empty
changelog.2.mfs  changelog.5.mfs  metadata.mfs.back    stats.mfs
changelog.3.mfs  changelog.6.mfs  metadata.mfs.back.1
[root@server1 mfs]# mv metadata.mfs.back metadata.mfs
[root@server1 mfs]# systemctl start moosefs-master
[root@server1 mfs]# systemctl stop moosefs-master.service

在这里插入图片描述

数据挂载过来服务可以启动表示没问题：

四、结合pacemaker部署高可用

我们的目标是创建一个master的集群，主机上设置vip，当某个master主机故障时，vip可以随之迁移，从而保证整个系统的正常运作，我们采用linux集群资源管理器pacemaker来实现这样的功能

安装

[root@server1 yum.repos.d]# cat dvd.repo       %配置仓库如下，使用rhel7.6镜像
[dvd]
name=rhel7.6
baseurl=http://172.25.0.250/rhel7.6
gpgcheck=0


[HighAvailability]
name=HighAvailability
baseurl=http://172.25.0.250/rhel7.6/addons/HighAvailability
gpgcheck=0

yum install -y pacemaker pcs psmisc policycoreutils-python     %安装相关组件

在这里插入图片描述

为了操作方便可以做server1和server4之间的免密：
ssh-keygen
ssh-copy-id server4

在这里插入图片描述

注意：要先保证两个节点上的防火墙和selinux都关闭

整体部署

// An highlighted block
var foo = 'bar';

在这里插入图片描述

corosync传递心跳信息：

pcs cluster start --all    %启动集群
pcs status          %查看集群状态
crm_verify  -LV     %检查集群配置

在这里插入图片描述

资源加入集群

接下来我们要在集群中加入mfs资源，实现一个mfs master高可用系统

注意：chunk server节点在更改完解析后记得重启服务

极端情况模拟

当出现更严重程度的损伤，比如内核崩溃，这时候我们该如何恢复集群状态呢？

五、启用fence机制

生产环境中可能会出现集群中某台节点由于网络闪断等原因导致某台节点处于假死状态，此时集群内部为了维持正常运作可能会出现资源的自动迁移，但是像mfs这种分布式存储系统，我们结合iscsi实现了存储共享，它是本地文件系统，因此不支持同时写入。这时候两个节点上都有资源，同时写入存储则会造成脑裂现象，这时候我们就需要一种防御机制，当出现这种情况时自动使假死状态的节点断电重启从而来防止脑裂现象的产生，维持集群的正常运作，一般fence是外界的物理设备，但是因为我们用的是虚拟机，所以我们可以使用libvirtd虚拟机管理工具来实现一个类似于fence的功能

服务端配置

在这里插入图片描述

[root@foundation0 ~]# fence_virtd -c
Module search path [/usr/lib64/fence-virt]: 

Available backends:
    libvirt 0.3
Available listeners:
    multicast 1.2

Listener modules are responsible for accepting requests
from fencing clients.

Listener module [multicast]: 

The multicast listener module is designed for use environments
where the guests and hosts may communicate over a network using
multicast.

The multicast address is the address that a client will use to
send fencing requests to fence_virtd.

Multicast IP Address [225.0.0.12]: 

Using ipv4 as family.

Multicast IP Port [1229]: 

Setting a preferred interface causes fence_virtd to listen only
on that interface.  Normally, it listens on all interfaces.
In environments where the virtual machines are using the host
machine as a gateway, this *must* be set (typically to virbr0).
Set to 'none' for no interface.

Interface [br0]:               %第一次配置这里需要填none

The key file is the shared key information which is used to
authenticate fencing requests.  The contents of this file must
be distributed to each physical host and virtual machine within
a cluster.

Key File [/etc/cluster/fence_xvm.key]: 

Backend modules are responsible for routing requests to
the appropriate hypervisor or management layer.

Backend module [libvirt]: 

The libvirt backend module is designed for single desktops or
servers.  Do not use in environments where virtual machines
may be migrated between hosts.

Libvirt URI [qemu:///system]: 

Configuration complete.

=== Begin Configuration ===
fence_virtd {
	listener = "multicast";
	backend = "libvirt";
	module_path = "/usr/lib64/fence-virt";
}

listeners {
	multicast {
		key_file = "/etc/cluster/fence_xvm.key";
		address = "225.0.0.12";
		interface = "br0";
		family = "ipv4";
		port = "1229";
	}

}

backends {
	libvirt {
		uri = "qemu:///system";
	}

}

=== End Configuration ===
Replace /etc/fence_virt.conf with the above [y/N]? y

在这里插入图片描述

[root@foundation0 cluster]# dd if=/dev/urandom  of=fence_xvm.key bs=128 count=1    %生成Key，只有拥有key的主机才能对libvirtd发送命令对虚拟机进行操作
1+0 records in
1+0 records out
128 bytes copied, 7.9739e-05 s, 1.6 MB/s
[root@foundation0 cluster]# ll
total 4
-rw-r--r-- 1 root root 128 Apr 18 17:22 fence_xvm.key
[root@foundation0 cluster]# netstat -anulp|grep :1229
[root@foundation0 cluster]# systemctl  start fence_virtd     %一定要先生成key再启动服务
[root@foundation0 cluster]# netstat -anulp|grep :1229        %开启端口
udp        0      0 0.0.0.0:1229            0.0.0.0:*                           20554/fence_virtd

在这里插入图片描述

客户端配置

安装相关rpm包：
在这里插入图片描述

复制key到集群两个节点上：

将fence加入到集群中：
pcs stonith create vmfence fence_xvm pcmk_host_map="server1:demo1;server4:demo4" op monitor interval=60s         %做主机映射并设置每隔60秒监控

pcs property set stonith-enabled=true       %启用fence

在这里插入图片描述
此时内核崩溃，网卡丢失等硬件问题能够得以解决，fence机制的存在将会使故障主机自动断电而不会造成脑裂：