一、MFS介绍
1、MFS特性
-
1. Free(GPL)
-
2. 通用文件系统,不需要修改上层应用就可以使用
-
3.可以在线扩容,体系架构可伸缩性极强,部署简单。
-
4. 高可用,可设置任意的文件冗余程度(提供比 raid1+0 更高的冗余级别,而绝对不会影响读或写的性能,只会加速!)
-
5. 可回收在指定时间内删除的文件( “ 回收站 ” 提供的是系统级别的服务,不怕误操作了,提供类似 oralce 的闪回等高级 dbms 的即时回滚特性!)
-
6. 提供 netapp,emc,ibm 等商业存储的 snapshot 特性。(可以对整个文件甚至在正在写入的文件创建文件的快照)
-
7. google filesystem 的一个 c 实现。
-
8. 提供 web gui 监控接口。
-
9. 提高随机读或写的效率。
-
10. 提高海量小文件的读写效率。
2、MFS 文件系统结构
- 包含 4 种角色:
- 管理服务器 managing server (master):负责各个数据存储服务器的管理,文件读写调度,文件空间回收以及恢复.多节点拷贝
- 元数据日志服务器 Metalogger server(Metalogger):负责备份 master 服务器的变化日志文件,文件类型为changelog_ml.*.mfs,以便于在 master server 出问题的时候接替其进行工作。
- 数据存储服务器 data servers (chunkservers):负责连接管理服务器,听从管理服务器调度,提供存储空间,并为客户提供数据传输。
- 客户机挂载使用 client computers:通过 fuse 内核接口挂接远程管理服务器上所管理的数据存储服务器,看起来共享的文件系统和本地 unix 文件系统使用一样的效果。
3、MFS读写原理
3、下载地址
百度网盘链接: https://pan.baidu.com/s/1HdtEIAO37Ngrx_i1SRs4-g 提取码: h3nv
二、实验环境
rhel7.3 selinux、firewalld off
1、Host解析(五台)
[root@server1 ~]# vim /etc/hosts
10.10.10.1 server1
10.10.10.2 server2
10.10.10.3 server3
10.10.10.4 server4
10.10.10.250 dream(真机)
2、Yum源
pacemaker的源自己本地搭建或者用网络的源都可以!!!
[root@server1 ~]# vim /etc/yum.repos.d/yum.repo
[rhel7.3]
name=rhel7.3
baseurl=http://10.10.10.250/rhel7.3
gpgcheck=0
[pacemaker]
name=pacemaker
baseurl=http://10.10.10.250/pacemaker
gpgcheck=0
三、MFS环境安装
官方参考文档:https://moosefs.com/Content/Downloads/moosefs-3-0-users-manual.pdf
1、master server安装
(1)安装
[root@server1 ~]# rpm -ivh moosefs-cgi-3.0.100-1.rhsystemd.x86_64.rpm moosefs-cgiserv-3.0.100-1.rhsystemd.x86_64.rpm moosefs-master-3.0.100-1.rhsystemd.x86_64.rpm
(2)启动服务及图形
[root@server1 ~]# mfsmaster
[root@server1 ~]# mfscgiserv start
[root@server1 ~]# netstat -lntup|grep 9425
tcp 0 0 0.0.0.0:9425 0.0.0.0:* LISTEN 1028/python
(3)查看结果
http://10.10.10.1:9425
<1> 报错:
<2> 解决方案
加入mfsmaster到host解析中!!!
[root@server1 ~]# vim /etc/hosts
10.10.10.1 server1 mfsmaster
10.10.10.2 server2
10.10.10.3 server3
10.10.10.4 server4
10.10.10.250 dream
2、chunkserver安装(server2和server3)
(1)安装
[root@server2 ~]# rpm -ivh moosefs-chunkserver-3.0.100-1.rhsystemd.x86_64.rpm
(2)配置mfshdd.cfg
[root@server2 ~]# vim /etc/mfs/mfshdd.cfg ###在最后添加
/mnt/mfschunk1
[root@server2 ~]# mkdir /mnt/mfschunk1
[root@server2 ~]# chown mfs.mfs /mnt/mfschunk1
[root@server3 ~]# vim /etc/mfs/mfshdd.cfg
/mnt/mfschunk2
[root@server3 ~]# mkdir /mnt/mfschunk2
[root@server3 ~]# chown mfs.mfs /mnt/mfschunk2
(3)2台加入host解析mfsmaster
[root@server2 ~]# vim /etc/hosts
10.10.10.1 server1 mfsmaster
10.10.10.2 server2
10.10.10.3 server3
10.10.10.4 server4
10.10.10.250 dream
(4)启动并查看结果
[root@server2 ~]# mfschunkserver start
[root@server2 ~]# netstat -lntup|grep 9422
tcp 0 0 0.0.0.0:9422 0.0.0.0:* LISTEN 10287/mfschunkserve
[root@server2 ~]# lsof -i :9420
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mfschunks 10287 mfs 12u IPv4 33187 0t0 TCP server2:35710->server1:9420 (ESTABLISHED)
http://10.10.10.1:9425
3、client安装及配置
(1)安装
[root@dream ~]# rpm -ivh moosefs-client-3.0.100-1.rhsystemd.x86_64.rpm
(2)加入host解析
[root@dream ~]# vim /etc/hosts
10.10.10.1 server1 mfsmaster
10.10.10.2 server2
10.10.10.3 server3
10.10.10.4 server4
10.10.10.250 dream
(3)配置mfsmount.cfg
[root@dream ~]# mkdir -p /mnt/mfs
[root@dream ~]# vim /etc/mfs/mfsmount.cfg ###在末尾添加
/mnt/mfs
(4)挂载
[root@dream ~]# df
[root@dream ~]# mfsmount ###能看到挂载
[root@dream ~]# df|grep mfsmaster
mfsmaster:9421 40480768 2747072 37733696 7% /mnt/mfs
四、基础操作
1、数据存储操作
mfssetgoal -r +数字:表示保存几份数据!!!
[root@dream ~]# mkdir /mnt/mfs/test{1..2}
###查看默认保存几份数据
[root@dream ~]# mfsgetgoal /mnt/mfs/test1
/mnt/mfs/test1: 2
[root@dream ~]# mfsgetgoal /mnt/mfs/test2
/mnt/mfs/test2: 2
###设置为1份
[root@dream ~]# mfssetgoal -r 1 /mnt/mfs/test2
[root@dream ~]# mfsgetgoal /mnt/mfs/test2
/mnt/mfs/test2: 1
###查看文件信息
[root@dream ~]# cp /etc/passwd /mnt/mfs/test1/
[root@dream ~]# cp /etc/passwd /mnt/mfs/test2/
[root@dream ~]# mfsfileinfo /mnt/mfs/test1/passwd
[root@dream ~]# mfsfileinfo /mnt/mfs/test2/passwd
2、数据恢复操作
(1)默认保存时间
[root@dream~]# mfsgettrashtime /mnt/mfs/test1/ ###86400s
/mnt/mfs/test1/: 86400
(2)默认数据块大小
一个chunk是64M
[root@dream ~]# cd /mnt/mfs/test1/
[root@dream test1]# dd if=/dev/zero of=testfile bs=1M count=200
[root@dream test1]# mfsfileinfo testfile
(3)数据恢复
[root@dream test1]# rm -f testfile
###挂载MFSMETA文件系统
[root@dream test1]# mkdir /mnt/mfsmeta
[root@dream test1]# mfsmount -m /mnt/mfsmeta/
[root@dream test1]# cd /mnt/mfsmeta/trash/
[root@dream trash]# find -name "*testfile*"
./007/00000007|test1|testfile
[root@dream trash]# mv "./007/00000007|test1|testfile" undel/
[root@dream trash]# ls /mnt/mfs/test1/
passwd testfile
五、Mfs+Pacemaker
1、安装软件
[root@server1 ~]# yum install -y pacemaker corosync pcs
[root@server4 ~]# yum install -y pacemaker corosync pcs
2、server1和server4互相免密登陆
[root@server1 ~]# ssh-keygen ###一直回车
[root@server1 ~]# ssh-copy-id -i .ssh/id_rsa.pub server1
[root@server1 ~]# scp -r .ssh/ root@10.10.10.4:
###测试,不要密码即可成功
[root@server1 ~]# ssh server4
[root@server4 ~]# ssh server1
3、集群操作
(1)启动服务
[root@server1 ~]# systemctl start pcsd
[root@server1 ~]# systemctl enable pcsd
[root@server4 ~]# systemctl start pcsd
[root@server4 ~]# systemctl enable pcsd
(2)配置集群密码,进行节点间认证
用户名:hacluster(只能这个)
[root@server1 ~]# id hacluster
[root@server1 ~]# echo 1 |passwd --stdin hacluster
[root@server1 ~]# ssh server4 'echo 1 |passwd --stdin hacluster'
[root@server1 ~]# pcs cluster auth server1 server4
Username: hacluster
Password:
server4: Authorized
server1: Authorized
(3)初始化集群及启动
[root@server1 ~]# pcs cluster setup --name cluster server1 server4 ###初始化
[root@server1 ~]# pcs cluster start --all ###启动
(4)查看集群状态
[root@server1 ~]# pcs status
Cluster name: cluster
WARNING: no stonith devices and stonith-enabled is not false
Stack: corosync
Current DC: server1 (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
Last updated: Wed Oct 24 21:22:22 2018
Last change: Wed Oct 24 21:20:57 2018 by hacluster via crmd on server1
2 nodes configured
0 resources configured
Online: [ server1 server4 ] ###online状态
No resources
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
4、基本配置
(1)关闭stonith-enabled(默认不支持)
[root@server1 ~]# crm_verify -L -V
error: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
error: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
error: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
Errors found during check: config not valid
[root@server1 ~]# pcs property set stonith-enabled=false
[root@server1 ~]# crm_verify -L -V ###没报错就对了
(2)配置VIP
[root@server1 ~]# pcs resource create vip ocf:heartbeat:IPaddr2 ip=10.10.10.100 cidr_netmask=32 op monitor interval=30s
[root@server1 ~]# pcs status
Cluster name: cluster
Stack: corosync
Current DC: server1 (version 1.1.18-11.el7-2b07d5c5a9) - partition with quorum
Last updated: Wed Oct 24 21:38:27 2018
Last change: Wed Oct 24 21:38:04 2018 by root via cibadmin on server1
2 nodes configured
1 resource configured
Online: [ server1 server4 ]
Full list of resources:
vip (ocf::heartbeat:IPaddr2): Started server1 ###可以看看刚刚配置的resource
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
[root@server1 ~]# ip addr ###可以看到我们加入的VIP
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:12:fd:9c brd ff:ff:ff:ff:ff:ff
inet 10.10.10.1/24 brd 10.10.10.255 scope global eth0
valid_lft forever preferred_lft forever
inet 10.10.10.100/32 brd 10.10.10.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::5054:ff:fe12:fd9c/64 scope link
valid_lft forever preferred_lft forever
(3)mfs启动脚本(server1和server4)
[root@server1 ~]# vim /usr/lib/systemd/system/mfsd.service
[Unit]
Description=mfs
After=network.target
[Service]
Type=forking
ExecStart=/usr/sbin/mfsmaster -a
ExecStop=/usr/sbin/mfsmaster stop
PrivateTmp=true
[Install]
WantedBy=multi-user.target
[root@server1 ~]# systemctl daemon-reload
[root@server1 ~]# systemctl status mfsd.service
● mfsd.service - mfs
Loaded: loaded (/usr/lib/systemd/system/mfsd.service; disabled; vendor preset: disabled)
Active: inactive (dead)
5、配置共享存储iscsi
(1)给虚拟机添加一块磁盘(server2)
[root@server2 ~]# fdisk -l
Disk /dev/vda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
(2)安装iscsi
[root@server2 ~]# yum install -y install targetcli
[root@server2 ~]# systemctl start target
[root@server2 ~]# systemctl enable target
(3)创建iscsi磁盘
[root@server2 ~]# targetcli
/> /backstores/block create server2:stroage1 /dev/vda ###我加入的磁盘名字为vda
/> /iscsi create iqn.2018-10.com.example:server2
/> /iscsi/iqn.2018-10.com.example:server2/tpg1/acls create iqn.2018-10.com.example:server2key
/> /iscsi/iqn.2018-10.com.example:server2/tpg1/luns create /backstores/block/server2:stroage1
/> exit
(4)安装iscsi服务(server1和server4)
[root@server1 ~]# yum install -y iscsi*
[root@server1 ~]# systemctl start iscsid
[root@server1 ~]# systemctl enable iscsid
(5)进行发现(server1和server4)
[root@server1 ~]# iscsiadm -m discovery -t st -p 10.10.10.2
10.10.10.2:3260,1 iqn.2018-10.com.example:server2
[root@server1 ~]# iscsiadm -m node -l ###会报错
###加入钥匙
[root@server1 ~]# vim /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2018-10.com.example:server2key
[root@server1 ~]# systemctl restart iscsid
[root@server1 ~]# iscsiadm -m discovery -t st -p 10.10.10.2
[root@server1 ~]# iscsiadm -m node -l
###发现多了块盘即为成功,2处的名字相同
[root@server1 ~]# fdisk -l
Disk /dev/sdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
(6)测试
[root@server1 ~]# fdisk /dev/sdb ###分区
[root@server1 ~]# mkfs.ext4 /dev/sdb1 ###格式化
server4中发现sdb1:
[root@server4 ~]# cat /proc/partitions
major minor #blocks name
8 0 20971520 sda
8 1 204800 sda1
8 2 20765696 sda2
253 0 20250624 dm-0
253 1 512000 dm-1
8 16 20971520 sdb
[root@server4 ~]# partx -a /dev/sdb
[root@server4 ~]# cat /proc/partitions
major minor #blocks name
8 0 20971520 sda
8 1 204800 sda1
8 2 20765696 sda2
253 0 20250624 dm-0
253 1 512000 dm-1
8 16 20971520 sdb
8 17 20970496 sdb1
6、迁移到共享存储中
(1)安装
[root@server4 ~]# rpm -ivh moosefs-master-3.0.100-1.rhsystemd.x86_64.rpm
(2)数据迁移
[root@server1 ~]# mount /dev/sdb1 /mnt
[root@server1 ~]# cd /var/lib/mfs/
[root@server1 mfs]# ls
changelog.0.mfs changelog.4.mfs metadata.crc stats.mfs
changelog.1.mfs changelog.5.mfs metadata.mfs.back
changelog.2.mfs changelog.6.mfs metadata.mfs.back.1
changelog.3.mfs changelog.7.mfs metadata.mfs.empty
###把数据复制到共享存储中
[root@server1 mfs]# mfsmaster stop
[root@server1 mfs]# cp -rp * /mnt/
[root@server1 ~]# umount /mnt
[root@server1 ~]# mount /dev/sdb1 /var/lib/mfs/
[root@server1 ~]# ll /var/lib/mfs/
total 3636
-rw-r----- 1 mfs mfs 75 Oct 24 22:26 changelog.1.mfs
-rw-r----- 1 mfs mfs 37 Oct 24 21:14 changelog.2.mfs
-rw-r----- 1 mfs mfs 884 Oct 24 20:33 changelog.3.mfs
-rw-r----- 1 mfs mfs 188 Oct 24 19:57 changelog.4.mfs
-rw-r----- 1 mfs mfs 136 Oct 24 00:07 changelog.5.mfs
-rw-r----- 1 mfs mfs 1169 Oct 23 23:32 changelog.6.mfs
-rw-r----- 1 mfs mfs 267 Oct 23 22:57 changelog.7.mfs
-rw-r----- 1 mfs mfs 45 Oct 23 21:19 changelog.8.mfs
-rw-r----- 1 mfs mfs 120 Oct 24 22:39 metadata.crc
-rw-r----- 1 mfs mfs 3992 Oct 24 22:39 metadata.mfs
-rw-r----- 1 mfs mfs 3992 Oct 24 22:00 metadata.mfs.back.1
-rw-r--r-- 1 mfs mfs 8 Jan 25 2018 metadata.mfs.empty
-rw-r----- 1 mfs mfs 3672832 Oct 24 22:39 stats.mfs
###授权并卸载
[root@server1 ~]# chown -R mfs.mfs /var/lib/mfs/
[root@server1 ~]# umount /var/lib/mfs/
(3)查看结果
[root@server4 ~]# mount /dev/sdb1 /var/lib/mfs/
[root@server4 ~]# ll /var/lib/mfs/
total 3636
-rw-r----- 1 mfs printadmin 75 Oct 24 22:26 changelog.1.mfs
-rw-r----- 1 mfs printadmin 37 Oct 24 21:14 changelog.2.mfs
-rw-r----- 1 mfs printadmin 884 Oct 24 20:33 changelog.3.mfs
-rw-r----- 1 mfs printadmin 188 Oct 24 19:57 changelog.4.mfs
-rw-r----- 1 mfs printadmin 136 Oct 24 00:07 changelog.5.mfs
-rw-r----- 1 mfs printadmin 1169 Oct 23 23:32 changelog.6.mfs
-rw-r----- 1 mfs printadmin 267 Oct 23 22:57 changelog.7.mfs
-rw-r----- 1 mfs printadmin 45 Oct 23 21:19 changelog.8.mfs
-rw-r----- 1 mfs printadmin 120 Oct 24 22:39 metadata.crc
-rw-r----- 1 mfs printadmin 3992 Oct 24 22:39 metadata.mfs
-rw-r----- 1 mfs printadmin 3992 Oct 24 22:00 metadata.mfs.back.1
-rw-r--r-- 1 mfs printadmin 8 Jan 25 2018 metadata.mfs.empty
-rw-r----- 1 mfs printadmin 3672832 Oct 24 22:39 stats.mfs
[root@server4 ~]# systemctl start mfsd
[root@server4 ~]# netstat -lntup
(4)关闭,把操作交给集群管理
[root@server4 ~]# systemctl stop mfsd
[root@server4 ~]# umount /var/lib/mfs/
7、resource和脚本添加
(1)resource添加
[root@server1 ~]# pcs resource create mfsDream ocf:heartbeat:Filesystem device="/dev/sdb1" directory="/var/lib/mfs/" fstype="ext4"
[root@server1 ~]# pcs status
(2)添加启动脚本
[root@server1 ~]# pcs resource create mfsd systemd:mfsd op monitor interval=30s
[root@server1 ~]# pcs resource standards
lsb
ocf
service
systemd
[root@server1 ~]# pcs status
(3)添加到同一个组
[root@server1 ~]# pcs resource group add mfsgroup vip mfsDream mfsd
[root@server1 ~]# pcs status
(4)修改host
[root@server1 ~]# vim /etc/hosts ###4台都修改
10.10.10.1 server1
10.10.10.2 server2
10.10.10.3 server3
10.10.10.4 server4
10.10.10.250 dream
10.10.10.100 mfsmaster
(5)重启chunkserver
[root@server2 ~]# mfschunkserver stop && mfschunkserver
[root@server3 ~]# mfschunkserver stop && mfschunkserver
(6)测试
在写入过程中,关闭服务进行测试!!!
[root@dream ~]# dd if=/dev/zero of=/mnt/mfs/test1/testfile bs=1M count=2000
[root@server1 ~]# pcs cluster standby server4
[root@server1 ~]# pcs status
[root@server1 ~]# pcs cluster unstandby server4 ###启动server4
8、fence配置
fence的操作可参考博客链接:https://blog.csdn.net/Dream_ya/article/details/80156603
(1)生成fence_xvm.key
[root@dream ~]# yum install -y fence-*
[root@dream ~]# mkdir /etc/cluster
[root@dream ~]# dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=128 count=1
(2)发送key
[root@server1 ~]# yum install -y fence-virt
[root@server4 ~]# yum install -y fence-virt
[root@server1 ~]# mkdir /etc/cluster
[root@server4 ~]# mkdir /etc/cluster
[root@dream ~]# scp /etc/cluster/fence_xvm.key root@10.10.10.1:/etc/cluster/
[root@dream ~]# scp /etc/cluster/fence_xvm.key root@10.10.10.4:/etc/cluster/
(3)Packet hash strength
[root@dream ~]# fence_virtd -c
[root@dream ~]# systemctl restart fence_virtd
[root@dream ~]# netstat -lntup|grep 1229
udp 0 0 0.0.0.0:1229 0.0.0.0:* 20133/fence_virtd
(4)启动stonith
[root@server1 ~]# pcs property set stonith-enabled=true
(5)配置fence
[root@server1 ~]# pcs stonith create mfsfence fence_xvm pcmk_host_map="server1:server1;server4:server4" op monitor interval=30s
[root@server1 ~]# pcs status
(6)设置为开机自启
[root@server1 ~]# systemctl enable {corosync,pacemaker}
[root@server4 ~]# systemctl enable {corosync,pacemaker}
[root@server1 ~]# pcs status
(7)测试
server1自动重启即为成功!!!
[root@server1 ~]# echo c >/proc/sysrq-trigger