配置:
1、两台一样配置的虚拟机,且在同一虚拟化平台上。
2、GlusterFS是单brick的volume,nfs server直接由其中一台虚拟机提供
# mount -t glusterfs master1:/k8s-pv2 /mnt/glusterfs
# mount -t nfs master1:/nfs-lv /mnt/nfs
说明:
iflag/oflag 提供 direct 模式,direct 模式是把写入请求直接封装成 I/O 指令发到磁盘,非 direct 模式只是把数据写入到系统缓存就认为 I/O 成功,并由操作系统决定缓存中的数据什么时候被写入磁盘。
写测试:
通过dd命令测试,写400M左右的数据到挂载目录下的文件,统计消耗时间
GlusterFS
#每次写入4K
# time dd if=/dev/zero of=/mnt/glusterfs/io bs=4k count=100000
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 41.9082 s, 9.8 MB/s
real 0m41.919s
user 0m0.628s
sys 0m5.992s
#每次写入1M
# time dd if=/dev/zero of=/mnt/glusterfs/io bs=1M count=410
410+0 records in
410+0 records out
429916160 bytes (430 MB) copied, 3.90263 s, 110 MB/s
real 0m4.002s
user 0m0.001s
sys 0m0.548s
#直接IO,每次写入4k
#time dd if=/dev/zero of=/mnt/glusterfs/io bs=4k count=100000 oflag=direct
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 39.2044 s, 10.4 MB/s
real 0m39.216s
user 0m0.942s
sys 0m5.005s
real 2m2.947s
user 0m0.047s
sys 0m3.440s
#直接IO,每次写入1M
# time dd if=/dev/zero of=/mnt/glusterfs/io bs=1M count=410 oflag=direct
410+0 records in
410+0 records out
429916160 bytes (430 MB) copied, 4.21479 s, 102 MB/s
real 0m4.227s
user 0m0.004s
sys 0m0.143s
GlusterFS iotop实时IO截图
直接IO,每次写入4k
直接IO,每次写入1M
NFS
#每次写入4k
# time dd if=/dev/zero of=/mnt/nfs/io bs=4k count=100000
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 5.18821 s, 78.9 MB/s
real 0m5.290s
user 0m0.174s
sys 0m0.884s
#每次写入1M
# time dd if=/dev/zero of=/mnt/nfs/io bs=1M count=410
410+0 records in
410+0 records out
429916160 bytes (430 MB) copied, 4.76596 s, 90.2 MB/s
real 0m4.924s
user 0m0.001s
sys 0m0.600s
#直接IO,每次写入4k
# time dd if=/dev/zero of=/mnt/nfs/io bs=4k count=100000 oflag=direct
100000+0 records in
100000+0 records out
409600000 bytes (410 MB) copied, 278.18 s, 1.5 MB/s
real 4m38.189s
user 0m0.188s
sys 0m3.608s
#直接IO,每次写入1M
#time dd if=/dev/zero of=/mnt/nfs/io bs=1M count=410 oflag=direct
410+0 records in
410+0 records out
429916160 bytes (430 MB) copied, 8.15538 s, 52.7 MB/s
real 0m8.164s
user 0m0.010s
sys 0m0.159s
NFS iotop实时IO截图
直接IO,每次写入4k
直接IO,每次写入1M
写测试结果:
经过多次直接IO写入测试,GlusterFS在大文件和小文件写入速度都远高于nfs。
在实际业务中会考虑到磁盘缓存,nfs的小文件写入速度远高于GlusterFS,是单brick的GlusterFS的10倍,当然GlusterFS可以做成条带卷来提高写的性能。
读测试:
通过dd命令测试,从挂载目录下的文件中读400M左右的数据,统计消耗时间。每次读前执行缓存回收
sync && echo 3 > /proc/sys/vm/drop_caches
GlusterFS
# time dd if=/mnt/glusterfs/io of=/dev/null bs=4k count=100
100+0 records in
100+0 records out
409600 bytes (410 kB) copied, 0.0959273 s, 4.3 MB/s
real 0m0.120s
user 0m0.001s
sys 0m0.005s
# time dd if=/mnt/glusterfs/io of=/dev/null bs=1M count=400
390+1 records in
390+1 records out
409600000 bytes (410 MB) copied, 9.50816 s, 43.1 MB/s
real 0m9.518s
user 0m0.006s
sys 0m0.344s
# time dd if=/mnt/glusterfs/io of=/dev/null bs=4k count=100 iflag=direct
100+0 records in
100+0 records out
409600 bytes (410 kB) copied, 0.262909 s, 1.6 MB/s
real 0m0.285s
user 0m0.008s
sys 0m0.005s
# time dd if=/mnt/glusterfs/io of=/dev/null bs=1M count=400 iflag=direct
390+1 records in
390+1 records out
409600000 bytes (410 MB) copied, 5.38114 s, 76.1 MB/s
real 0m5.391s
user 0m0.000s
sys 0m0.041s
NFS
# time dd if=/mnt/nfs/io of=/dev/null bs=4k count=100
100+0 records in
100+0 records out
409600 bytes (410 kB) copied, 0.0703449 s, 5.8 MB/s
real 0m0.076s
user 0m0.002s
sys 0m0.005s
# time dd if=/mnt/nfs/io of=/dev/null bs=1M count=400
400+0 records in
400+0 records out
419430400 bytes (419 MB) copied, 3.69 s, 114 MB/s
real 0m3.717s
user 0m0.003s
sys 0m0.350s
# time dd if=/mnt/nfs/io of=/dev/null bs=4k count=100 iflag=direct
100+0 records in
100+0 records out
409600 bytes (410 kB) copied, 0.0862116 s, 4.8 MB/s
real 0m0.095s
user 0m0.000s
sys 0m0.011s
# time dd if=/mnt/nfs/io of=/dev/null bs=1M count=400 iflag=direct
400+0 records in
400+0 records out
419430400 bytes (419 MB) copied, 4.24663 s, 98.8 MB/s
real 0m4.265s
user 0m0.001s
sys 0m0.103s
读测试结果:
经过多次读测试,NFS在大文件或者小文件的读性能上都优于GlusterFS