glusterfs文件系统性能测试

不同挂载模式在不同块大小之间六种读写方式的速率测试

  概要

挂载模式:Nfs Gluster Fuse Local(本地) 
BS:4k 16k 64k 256k 1M 4M
读写方式:randread read randwrite write randrw readwrite
测试计划:
   每次测试10分钟,一组要6个小时,共四组
   比较它们之间的差别和性能差异
测试环境:
   虚拟机一台-192.168.0.134 (通过远程登录进行操作)

  一. 搭建环境,安装软件

• 1.1 安装软件
  • #yum install glusterfs-server
  • #yum install qemu
• 1.2 启动服务
  • #service glusterd start
  • #service glusterd status

    Redirecting to /bin/systemctl status glusterd.service 
    glusterd.service - GlusterFS an clustered file-system server 
    Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled) 
    Active: active (running) since 五 2014-01-06 19:30:17 CST; 5 days ago 
    Main PID: 735 (glusterd) CGroup: name=systemd:/system/glusterd.service 
    ├─ 735 /usr/sbin/glusterd -p /run/glusterd.pid 
    ├─ 745 /usr/sbin/glusterfsd -s 192.168.0.134 --volfile-id test.192... 
    └─1066 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -... 

  二. create and config volume

• 2.1 在/home/test目录下创建volume test
  • #gluster volume create test 192.168.0.134:/home/test
• 2.2 设置volume并查看它的信息
  • #gluster volume set test nfs.disable off
  • #gluster volume info

    Volume Name: test 
    Type: Distribute 
    Volume ID: 62ddcf55-7aba-4bc9-bd96-3a776118c137 
    Status: Started 
    Number of Bricks: 1 
    Transport-type: tcp 
    Bricks: 
    Brick1: 192.168.0.134:/home/test 
    Options Reconfigured: 
    nfs.ports-insecure: 1 
    nfs.disable: off 
    auth.allow: all 
    nfs.rpc-auth-allow: * 

  三. create a raw image

• #cd /home/test
• #qemu-img create -f raw test.img 5G
• #ls

  test.img

  四. mount volume

• 4.1 NFS
  • #mount -t nfs -o vers=3 192.168.0.134:/test /mnt/test
• 4.2 Fuse
  • #mount -t glusterfs 192.168.0.134:/test /mnt/test
• 4.3 Gluster&Local
  • Gluster,Local don't mount

  五. start qemu service

• 5.1 NFS
  • #qemu-system-x86_64 -machine accel=kvm -cpu host -m 1024 -nodefaults -nographic -drive file=fedora19.qcow2,if=none,id=drive0 -device virtio-blk-pci,drive=drive0,id=disk0,bootindex=1 -netdev user,id=net0,hostfwd=tcp::7022-:22 -device virtio-net-pci,netdev=net0,id=net0 -drive file=/home/test/test.img,if=none,id=drive1 -device virtio-blk-pci,drive=drive1,id=disk1,bootindex=2

• 5.2 Fuse
  • #qemu-system-x86_64 -machine accel=kvm -cpu host -m 1024 -nodefaults -nographic -drive file=fedora19.qcow2,if=none,id=drive0 -device virtio-blk-pci,drive=drive0,id=disk0,bootindex=1 -netdev user,id=net0,hostfwd=tcp::7022-:22 -device virtio-net-pci,netdev=net0,id=net0 -drive file=/mnt/test/test.img,if=none,id=drive1 -device virtio-blk-pci,drive=drive1,id=disk1,bootindex=2

• 5.3 Gluster
  • #qemu-system-x86_64 -machine accel=kvm -cpu host -m 1024 -nodefaults -nographic -drive file=fedora19.qcow2,if=none,id=drive0 -device virtio-blk-pci,drive=drive0,id=disk0,bootindex=1 -netdev user,id=net0,hostfwd=tcp::7022-:22 -device virtio-net-pci,netdev=net0,id=net0 -drive file=gluster://192.168.0.134/test/test.img,if=none,id=drive1 -device virtio-blk-pci,drive=drive1,id=disk1,bootindex=2

• 5.4 Local
  • #qemu-system-x86_64 -machine accel=kvm -cpu host -m 1024 -nodefaults -nographic -drive file=fedora19.qcow2,if=none,id=drive0 -device virtio-blk-pci,drive=drive0,id=disk0,bootindex=1 -netdev user,id=net0,hostfwd=tcp::7022-:22 -device virtio-net-pci,netdev=net0,id=net0 -drive file=/home/test/test.img,if=none,id=drive1 -device virtio-blk-pci,drive=drive1,id=disk1,bootindex=2

  六. 数据的接收

• 6.1 远程通过端口登录当前主机
  • #ssh root@192.168.0.134 -p 7022
• 6.2 使用fio测速工具
  • #sudo yum install fio
• 6.3 编写脚本进行测试
  • # cat test.sh 
    

    #!/bin/sh
    export testing=nfs
    for j in `echo -e "read\
    write\
    readwrite\
    randread\
    randwrite\
    randrw"`;
        do for i in `echo -e "4k\
    16k\
    64k\
    256k\
    1024k\
    4096k"`;
            do fio -filename=/dev/vdb -direct=1 -rw=$j -bs=$i -size 8G -numjobs=8 -runtime=600 -group_reporting -name=$testing-$j-$i --output=$testing/$testing-$j-$i
        done;
    done;
    

  七. 数据分析

• 7.1 测试结果对比(横向表示六种读写方式的块大小,竖向表示六种读写模式的iops/bw/lat,单位bw:KB/s,clat:msec)

    NFS

测试对象NFS	4k	16k	64k	256k	1024k	4096k
read iops	989	778	671	312	66	20
write iops	642	606	389	259	116	19
randwrite iops	311	231	105	44	14	3
randread iops	523	232	87	35	15	4
readwrite iops	471	297	218	107	28	6
randrw iops	150	112	56	19	7	2
read bw	3958	12453	42997	80027	67799	84194
write bw	2569	9696	24907	66541	119650	81431
randwrite bw	1254	3704	6783	11278	14825	15071
randread bw	2092	3720	5626	9015	15633	18789
readwrite bw	1889	4760	14000	27450	29228	27939
randrw bw	620	1810	3580	5090	7985	8563
read clat	8.0	10.2	11.8	25.4	120.66	388.59
write clat	12.3	13.0	20.4	30.5	67.43	396.71
randwrite clat	25.4	34.3	75.17	181.19	550.47	2166.38
randread clat	15.1	34.2	90.8	226.91	523.69	1741.25
readwrite clat	7.9	12.9	16.9	34.6	129.1	224.30
randrw clat	24.9	35.5	93.2	230.85	504.18	2077.93

     Gluster

测试对象Gluster	4k	16k	64k	256k	1024k	4096k
read iops	344	335	160	164	63	29
write iops	559	253	181	122	49	26
randwrite iops	330	217	98	40	13	3
randread iops	146	187	186	87	20	6
readwrite iops	330	217	98	40	13	3
randrw iops	77	84	57	25	6	1
read bw	1377	5372	10262	41988	65066	121444
write bw	2239	4056	11599	31234	50361	109998
randwrite bw	1321	3486	6313	10333	14088	14452
randread bw	599	2998	11940	22512	21041	26648
readwrite bw	1321	3486	6313	10333	14088	14452
randrw bw	318	1359	3694	6539	7079	5932
read clat	23.1	23.7	49.76	48.6	125.79	269.48
write clat	14.22	31.40	43.94	65.18	160.85	292.82
randwrite clat	24.09	36.57	80.88	197.44	576.46	2234.40
randread clat	54.55	42.60	42.81	90.87	389.14	1228.36
readwrite clat	21.70	27.32	38.29	58.56	128.57	587.20
randrw clat	92.24	83.47	110.60	181.55	633.95	2959.76

     Fuse

测试对象Fuse	4k	16k	64k	256k	1024k	4096k
read iops	694	606	506	244	79	30
write iops	413	245	188	103	37	11
randwrite iops	224	154	102	44	13	3
randread iops	147	194	147	32	19	5
readwrite iops	288	165	151	77	22	7
randrw iops	70	70	35	18	7	1
read bw	2779	9701	32405	62628	81854	125516
write bw	1653	3935	12077	26528	38646	47418
randwrite bw	918	2475	6540	11275	13600	14760
randread bw	603	3106	9460	8278	19555	24226
readwrite bw	1157	2670	9711	19944	23381	32197
randrw bw	290	1130	2286	4941	7500	7687
read clat	11.4	13.1	15.7	32.6	99.9	260.69
write clat	19.26	32.4	42.2	76.70	210.84	686.93
randwrite clat	35.5	51.5	78.06	181.31	600.94	2207.51
randread clat	54.2	41.0	54.0	247.25	418.71	1351.66
readwrite clat	10.3	21.0	21.07	40.8	141.06	122.04
randrw clat	99.0	96.9	196.9	275.0	558.38	2313.34

     Local

测试对象Local	4k	16k	64k	256k	1024k	4096k
read iops	639	597	560	274	68	20
write iops	580	473	358	215	111	6
randwrite iops	497	154	97	41	12	3
randread iops	493	273	82	45	15	4
readwrite iops	360	247	186	102	26	6
randrw iops	154	126	37	17	6	2
read bw	2556	9560	35843	70385	69662	83230
write bw	2321	7574	22970	55174	114601	25056
randwrite bw	1991	2474	6258	10683	13233	15291
randread bw	1972	4379	5293	11690	15707	18752
readwrite bw	1442	4000	12000	26184	26783	27208
randrw bw	638	2020	2423	4400	6956	8685
read clat	12.4	13.3	14.2	28.9	117.41	339.10
write clat	13.6	16.7	22.1	36.8	70.44	1297.42
randwrite clat	15.8	51.4	81.4	191.19	616.78	2131.45
randread clat	16.1	29.1	96.5	175.0	521.24	1746.24
readwrite clat	10.5	15.2	20.4	36.2	135.99	229.13
randrw clat	27.7	37.7	118.4	288.6	533.1	1990.0

• 7.2 结果分析
  • 7.2.1 随着bs越大，bw越大
  • 7.2.2 四种模式randrw的iops和bw小于其它读写方式，不再一个数量级上
  • 7.2.3 NFS与Gluster比较: NFS的clat小，bw大，iops也更快 (在大部分情况下)
• 7.3 性能比较

     randread

randread
4k	nfs>local>fuse>gluster
16k	local>nfs>fuse>gluster
64k	gluster>fuse>nfs>local
256k	gluster>local>nfs>fuse
1024k	gluster>fuse>local>nfs
4M	gluster>fuse>nfs>local

     randrw

randrw
4k	local>nfs>gluster>fuse
16k	local>nfs>gluster>fuse
64k	gluster>nfs>local>fuse
256k	gluster>nfs>fuse>local
1024k	nfs>fuse>gluster>local
4M	local>nfs>fuse>gluster

     randwrite

randwrite
4k	local>gluster>nfs>fuse
16k	nfs>gluster>fuse>local
64k	nfs>fuse>gluster>local
256k	nfs>fuse>local>gluster
1024k	nfs>gluster>fuse>local
4M	local>nfs>fuse>gluster

     read

read
4k	nfs>local>fuse>gluster
16k	nfs>fuse>local>gluster
64k	nfs>local>fuse>gluster
256k	nfs>local>fuse>gluster
1024k	fuse>local>nfs>gluster
4M	fuse>gluster>nfs>local

     readwrite

readwrite
4k	nfs>local>fuse>gluster
16k	nfs>local>fuse>gluster
64k	nfs>local>fuse>gluster
256k	nfs>local>fuse>gluster
1024k	gluster>nfs>local>fuse
4M	fuse>nfs>local>gluster

     write

write
4k	nfs>local>gluster>fuse
16k	nfs>local>gluster>fuse
64k	nfs>local>fsue>gluster
256k	nfs>local>gluster>fuse
1024k	nfs>local>gluster>fuse
4M	gluster>nfs>local>fuse

  八. 可能遇到的问题及其解决办法

• volume create: test: failed: /home/test or a prefix of it is already part of a volume

#setfattr -x trusted.glusterfs.volume-id /home/test 
#setfattr -x trusted.gfid /home/test

• mount.nfs: requested NFS version or transport protocol is not supported

#gluster volume set test nfs.ports-insecure 1 
#gluster volume stop test gluster volume start test

• config volume

Options Reconfigured:

nfs.disable: off 
auth.allow: all 
nfs.rpc-auth-allow: * 

• portmap mountd nfsd statd lockd rquotad:192.168.0.*

#systemctl start portmap.service

• volume start: test: failed: Volume id mismatch for brick 192.168.0.134:/home/test. Expected volume id 5c110d50-4d8b-4f2a-af09-3456ec11f26b, volume id 2b3ebaea-8517-4259-84df-3d287f53cad5 found

#dhclient 
#mount -t ext4 /dev/vdb1 /home/test

• Connection failed. Please check if gluster daemon is operational

#systemctl start glusterd.service

• [root@localhost ~]# mount -t nfs -o vers=3 192.168.0.134:/test /home/test/

mount.nfs: rpc.statd is not running but is required for remote locking. 
mount.nfs: Either use '-o nolock' to keep locks local, or start statd. 
mount.nfs: an incorrect mount option was specified 

[root@localhost ~]# systemctl start rpcbind.service

• 如果在执行脚本时遇到错误提示，例如 no such file/directory

将脚本中的fio参数项-direct=1去掉，然后在执行

  九. 名词解析

io=		Number of megabytes io performed
bs=		Block size
rw=		Type of io psttern
bw=		Bandwidth rate
iops=           IOs performed per second
runt=		The runtime of that thread
direct=bool	If value is true, use non-buffered io
iodepth=int	This defines how many io units to keep in flight against
		the file
numjobs=int	Create the specified number of clones of this job(并发数)
clat=		Completion latency