容量、聚合带宽、延迟:
这三个东东,容量看起来不复杂,就是虚拟存储池,把所有的可使用的逻辑分区+服务器IP地址 组合以下/ 然后形成要给超级大的总存储;聚合带宽,这个关键时聚合之间是如何平衡和保持HA的,如果大家多是连接单独的点//存储层应该不是直连的,只有到了类似上层的应用,才会是直连的//
MooseFS使用FUSE (Filesystem in User Space) 实现文件系统挂载,能用于所有支持Fuse的操作系统(Linux, FreeBSD, Mac OS X等),它还支持特殊的文件(块设备,字符设备和管道),支持软链接,支持快照,回收站等,每个目录的冗余策略可配置,具有一个完善的监控界面。目前在国内互联网公司得到广泛使用,例如豆瓣网使用MooseFS存储大量图片,遨游使用MooseFS存储用户的收藏夹。
2Calls to mfsmaster(named as master, not meta…),1:where 8:write status。从这个图也说明了,客户端写的时候应该在两种情况下触发:
1.客户端文件要发送network IO,已经达到一个block的大小了,比如64M
2.客户端文件写已经结束了,那么再小的内容,也要写到一个block
mfs client
3.1 安装fuse
yum install kernel.x86_64 kernel-devel.x86_64 kernel-headers.x86_64
###reboot server####
yum install fuse.x86_64 fuse-devel.x86_64 fuse-libs.x86_64
modprobe fuse
--
3.2 安装mfsclient
wget http://ncu.dl.sourceforge.net/project/moosefs/moosefs/1.6.11/mfs-1.6.11.tar.gz
tar zxvf mfs-1.6.11.tar.gz
cd mfs-1.6.11
useradd mfs -s /sbin/nologin
./configure --prefix=/usr/local/mfs --with-default-user=mfs --with-default-group=mfs --enable-mfsmount
make
make install
3.3 挂载文件系统
cd /mnt/
mkdir mfs
/usr/local/mfs/bin/mfsmount /mnt/mfs/ -H 192.168.28.242
mkdir mfsmeta
/usr/local/mfs/bin/mfsmount -m /mnt/mfsmeta/ -H 192.168.28.242
df -ah
====〉客户端为什么需要有这个 mfsmeta这个 目录挂载过来?
系统管理
4.1 管理命令
设置副本 的份数,推荐3份
/usr/local/mfs/bin/mfssetgoal -r 3 /mnt/mfs
查看某文件
/usr/local/mfs/bin/mfsgetgoal /mnt/mfs
查看目录信息
/usr/local/mfs/bin/mfsdirinfo -H /mnt/mfs
5. 性能测试---看这些基本数据留作比较
5.1 mfs
1. 大文件(block=1M byte)
dd if=/dev/zero of=1.img bs=1M count=5000
5242880000 bytes (5.2 GB) copied, 48.8481 seconds, 107 MB/s
2. 小文件( 50 byte * 100w个 * 1 client ) ( 1000 * 1000 ) 写入
real 83m41.343s
user 4m17.993s
sys 16m58.939s
列表
time find ./ -type f | nl | tail
999999 ./0/1
1000000 ./0/0
real 0m39.418s
user 0m0.721s
sys 0m0.225s
删除
time rm -fr *
real 6m35.273s
user 0m0.394s
sys 0m23.546s
3. 小文件( 1K byte * 100w个 * 100 client ) { 1000 * 1000 )
写入(100client)
time ../../p_touch_file.sh
real 22m51.159s
user 4m42.850s
sys 18m41.437s
列表(1client)
time find ./ | nl | tail
real 0m35.910s
user 0m0.628s
sys 0m0.204s
删除(1client)
time rm -fr *
real 6m36.530s
user 0m0.697s
sys 0m21.682s
4. 小文件(1k byte* 100w个 * 200 client) { 1000 * 1000 )
time ../../p_touch_file.sh
real 27m56.656s
user 5m12.195s
sys 20m52.079s
5. 小文件(1k byte* 100w个 * 1000 client) { 1000 * 1000 )
写入
time ../../p_touch_file.sh
real 30m30.336s
user 5m6.607s
sys 21m
5.2 本地磁盘
1. 大文件(block=1M byte)
dd if=/dev/zero of=1.img bs=1M count=5000
5242880000 bytes (5.2 GB) copied, 58.7371 seconds, 89.3 MB/s
2. 小文件(50 byte * 100w个 * 1 client) { 1000 * 1000 )
写入
time ../touch_file.sh
real 17m47.746s
user 4m54.068s
sys 12m54.425s
列表
time find ./ -type f | nl | tail
1000000 ./875/582
1000001 ./875/875
real 0m9.120s
user 0m1.102s
sys 0m0.726s
删除
time rm -fr *
real 0m37.201s
user 0m0.432s
sys 0m15.268s
两个Client同时dd测试
数据块1M 文件大小20G
Client1 写:68.4MB/s 读:25.3MB/s
Client2 写:67.5MB/s 读:24.7MB/s
总吞吐:写:135.9MB/s 读:50.0MB/s
写命令:dd if=/dev/zero of=/mfs/test.1 bs=1M count=20000
读命令:dd if=/mfs/test.1 of=/dev/null bs=1M
1.4 启动和停止web gui
启动: /usr/local/mfs/sbin/mfscgiserv
停止: kill /usr/local/mfs/sbin/mfscgiserv
2.1 从块设备创建本地文件系统
fdisk -l
mkfs.ext3 /dev/sdb
mkdir /data
chown mfs:mfs /data
mount -t ext3 /dev/sdb /data
df -ah
/dev/sdb 133G 188M 126G 1% /data
mfs nagios监控程序
#!/bin/bash
# ===========================================================================================================
#
# MFS status check plugins for nagios
#
# Written by : liu yunfeng
# Release : 1.0
# Create Date : 2010-05-18
# Description : Nagios plugins (script) to check MFS status
#
# ===========================================================================================================
# Nagios return codes
STATE_OK=0
STATE_WARNING=1
STATE_CRITICAL=2
STATE_UNKNOWN=3
#Get Master server Process
Master(){
MFS_PID=`ps -ef | grep mfsmaster | grep ${mfspath} | awk '{print $2}'`
if [ "${MFS_PID}" = "" ];then
echo "ERROR : mfs master does not start"
exit $STATE_CRITICAL
else
echo "OK : mfs master server running OK!"
exit $STATE_OK
fi
}
#Get chunker server Process
Chunker(){
MFS_PID=`ps -ef | grep mfschunkserver | grep ${mfspath} | awk '{print $2}'`
if [ "${MFS_PID}" = "" ];then
echo "ERROR : mfs chunkserver does not start"
exit $STATE_CRITICAL
else
echo "OK : mfs master chunkserver running OK!"
exit $STATE_OK
fi
}
#Get disk used status
Disk_used(){
Disk_used=`df -h | grep ${mountpoint} |awk '{print $4}'`
if [ ${Disk_used%\%} -ge ${warningdiskused} -a ${Disk_used%\%} -lt ${criticaldiskused} ];then
echo "Waring :mfs disk used Rate is "${Disk_used}""
exit $STATE_WARNING
elif [ ${Disk_used%\%} -ge ${criticaldiskused} ];then
echo "CRITICAL :mfs disk used Rate is "${Disk_used}""
exit $STATE_CRITICAL
elif [ "${Disk_used%\%}" = "" ];then
echo "CRITICAL :mfs client not mount on"
exit $STATE_CRITICAL
else
echo "OK : MFS client mount on and disk used rate is "${Disk_used}" , Everything is OK!"
exit $STATE_OK
fi
}
#Get metalogger server Process
Metalogger(){
MFS_PID=`ps -ef | grep mfsmetalogger | grep ${mfspath} | awk '{print $2}'`
if [ "${MFS_PID}" = "" ];then
echo "ERROR : mfs mfsmetalogger does not start"
exit $STATE_CRITICAL
else
echo "OK : mfs master mfsmetalogger running OK!"
exit $STATE_OK
fi
}
# Functions plugin usage
print_usage()
{
echo ""
echo " -M MFS master server status "
echo " -C MFS chunker server status "
echo " -U MFS client disk used Rate "
echo " -L MFS metalogger server status "
echo " -h Show this page "
echo ""
echo " Usage1 : check_mfs.sh -M mfsmasterpath"
echo " ex : check_mfs.sh -M /usr/local/mfs"
echo " Usage2 : check_mfs.sh -C mfschunkpath"
echo " ex : check_mfs.sh -C /usr/local/mfs"
echo " Usage3 : check_mfs.sh -U mountpoint warningrate criticalrate"
echo " ex : check_mfs.sh -U /mnt/mfs 60 80"
echo " Usage4 : check_mfs.sh -L mfsmetaloggerpath"
echo " ex : check_mfs.sh -L /usr/local/mfs"
echo " ex : check_mfs.sh -h"
echo ""
exit 0
}
# Parse parameter
while [ $# -gt 0 ]
do
case "$1" in
-h | --help)
print_usage
exit $STATE_OK
;;
-M | --master)
mfspath=${2:-"/usr/local/mfs"}
Master
exit
;;
-C | --chunker)
mfspath=${2:-"/usr/local/mfs"}
Chunker
;;
-U | --usedwarning)
mountpoint=${2:-"/mnt/mfs"}
warningdiskused=${3:-"75"}
criticaldiskused=${4:-"85"}
Disk_used
;;
-L | --metalogger)
mfspath=${2:-"/usr/local/mfs"}
Metalogger
;;
*) echo "Unknown argement: $1"
exit $STATE_UNKNOWN
;;
esac
shift
done