linux ext4 慢,性能 – 为什么ZFS比ext4和btrfs慢得多？

问题

我最近安装了一个新磁盘并在其上创建了一个zpool： /# zpool create morez /dev/sdb

使用它一段时间后,我注意到它很慢： /morez# fio --name rw --rw rw --size 10G read: IOPS=19.6k, BW=76.6MiB/s (80.3MB/s)(5120MiB/66834msec) write: IOPS=19.6k, BW=76.6MiB/s (80.3MB/s)(5120MiB/66834msec)

此测试与我的实际用例非常相似.我正在从磁盘读取中等数量(~10k)的图像(每个~2 MiB).当磁盘大部分为空时,它们都被立即写入,因此我不认为它们会被分割.

为了比较,我测试了ext4： /# gdisk /dev/sdb.../# mkfs.ext4 -f /dev/sdb1 && mount /dev/sdb1 /mnt && cd /mnt/mnt# fio --name rw --rw rw --size 10G read: IOPS=48.3k, BW=189MiB/s (198MB/s)(5120MiB/27135msec) write: IOPS=48.3k, BW=189MiB/s (198MB/s)(5120MiB/27135msec)

和btrfs： /# mkfs.btrfs -f /dev/sdb1 && mount /dev/sdb1 /mnt && cd /mnt/mnt# fio --name rw --rw rw --size 10G read: IOPS=51.3k, BW=201MiB/s (210MB/s)(5120MiB/25528msec) write: IOPS=51.3k, BW=201MiB/s (210MB/s)(5120MiB/25528msec)

可能导致ZFS性能问题的原因是什么？如何让它更快？

尝试解决方案失败

我还尝试显式设置zpool的扇区大小,因为我的磁盘(Seagate ST1000DM003)使用4096字节的物理扇区： /# zpool create -o ashift=12 morez /dev/sdb

这没有改善性能： /morez# fio --name rw --rw rw --size 10G read: IOPS=21.3k, BW=83.2MiB/s (87.2MB/s)(5120MiB/61573msec) write: IOPS=21.3k, BW=83.2MiB/s (87.2MB/s)(5120MiB/61573msec)

意见

奇怪的是,使用zvol有很好的表现： /# zfs create -V 20G morez/vol/# fio --name rw --filename /dev/zvol/morez/vol --rw rw --size 10G read: IOPS=52.7k, BW=206MiB/s (216MB/s)(5120MiB/24852msec) write: IOPS=52.7k, BW=206MiB/s (216MB/s)(5120MiB/24852msec)

为什么这只影响ZFS文件系统而不影响zvols？

btrfs的扩展测试

在评论中,有人认为差异可能是由于缓存造成的.经过进一步测试,我不相信是这种情况.我将btrfs测试的大小增加到远高于我的计算机的内存量,其性能仍远远高于ZFS： /# mkfs.btrfs -f /dev/sdb1 && mount /dev/sdb1 /mnt && cd /mnt/mnt# $fio --name rw --rw rw --size 500G --runtime 3600 --time_based --ramp_time 900 read: IOPS=41.9k, BW=164MiB/s (172MB/s)(576GiB/3600003msec) write: IOPS=41.9k, BW=164MiB/s (172MB/s)(576GiB/3600003msec)

系统信息

软件

> Arch Linux,内核版本4.11.6

> Linux 0.6.5.10上的ZFS

> fio 2.21

硬件

>正在测试的驱动器：Seagate ST1000DM003,连接到6Gb / s SATA端口

>主板：Gigabyte X99-SLI

>记忆：8 GiB

ZFS信息

以下是运行fio之前ZFS属性的样子.这些只是使用默认设置创建zpool的结果. # zpool get all morezNAME PROPERTY VALUE SOURCEmorez size 928G -morez capacity 0% -morez altroot - defaultmorez health ONLINE -morez guid [removed] defaultmorez version - defaultmorez bootfs - defaultmorez delegation on defaultmorez autoreplace off defaultmorez cachefile - defaultmorez failmode wait defaultmorez listsnapshots off defaultmorez autoexpand off defaultmorez dedupditto 0 defaultmorez dedupratio 1.00x -morez free 928G -morez allocated 276K -morez readonly off -morez ashift 0 defaultmorez comment - defaultmorez expandsize - -morez freeing 0 defaultmorez fragmentation 0% -morez leaked 0 defaultmorez feature@async_destroy enabled localmorez feature@empty_bpobj enabled localmorez feature@lz4_compress active localmorez feature@spacemap_histogram active localmorez feature@enabled_txg active localmorez feature@hole_birth active localmorez feature@extensible_dataset enabled localmorez feature@embedded_data active localmorez feature@bookmarks enabled localmorez feature@filesystem_limits enabled localmorez feature@large_blocks enabled local# zfs get all morezNAME PROPERTY VALUE SOURCEmorez type filesystem -morez creation Thu Jun 29 19:34 2017 -morez used 240K -morez available 899G -morez referenced 96K -morez compressratio 1.00x -morez mounted yes -morez quota none defaultmorez reservation none defaultmorez recordsize 128K defaultmorez mountpoint /morez defaultmorez sharenfs off defaultmorez checksum on defaultmorez compression off defaultmorez atime on defaultmorez devices on defaultmorez exec on defaultmorez setuid on defaultmorez readonly off defaultmorez zoned off defaultmorez snapdir hidden defaultmorez aclinherit restricted defaultmorez canmount on defaultmorez xattr on defaultmorez copies 1 defaultmorez version 5 -morez utf8only off -morez normalization none -morez casesensitivity sensitive -morez vscan off defaultmorez nbmand off defaultmorez sharesmb off defaultmorez refquota none defaultmorez refreservation none defaultmorez primarycache all defaultmorez secondarycache all defaultmorez usedbysnapshots 0 -morez usedbydataset 96K -morez usedbychildren 144K -morez usedbyrefreservation 0 -morez logbias latency defaultmorez dedup off defaultmorez mlslabel none defaultmorez sync standard defaultmorez refcompressratio 1.00x -morez written 96K -morez logicalused 72.5K -morez logicalreferenced 40K -morez filesystem_limit none defaultmorez snapshot_limit none defaultmorez filesystem_count none defaultmorez snapshot_count none defaultmorez snapdev hidden defaultmorez acltype off defaultmorez context none defaultmorez fscontext none defaultmorez defcontext none defaultmorez rootcontext none defaultmorez relatime off defaultmorez redundant_metadata all defaultmorez overlay off default