谈谈 Zfs_vdev_max_pending

Zfs_vdev_max_pending :controls the maximum number of concurrent I/Os pending to each device.

it's the queue depth to each vdev

• default is 10
• SAS - 10
• SATA - 4
• SSD - 32

you can go higher than 10 depending on how many spindles you have in the back, and what your latency target is

as queue depth increases, throughput increases but latency also increases

Default: 10 in 3.1.x

SATA disks do Native Command Queuing while SAS disks do Tagged Command Queuing.

Apparently OpenSolaris/Solaris is optimized for the latter with a 32 wide command queue set by default. This completely saturates the SATA disks with IO commands in turn making the system unusable for short periods of time. 

---

In a storage array where LUNs are made of a large number of disk drives, the ZFS queue can become a limiting factor on read IOPS. This behavior is one of the underlying reasoning for the best practice of presenting as many LUNS as there are backing spindles to the ZFS storage pool. That is, if you create LUNS from a 10 disk-wide array level raid-group, then using 5 to 10 LUNs to build a storage pool allows ZFS to manage enough of an I/O queue without the need to set this specific tunable.

However, when no separate intent log is in use and the pool is made of JBOD disks, using a small zfs_vdev_max_pending value, such as 10, can improve the synchronous write latency as those are competing for the disk resource. Using separate intent log devices can alleviate the need to tune this parameter for loads that are synchronously write intensive since those synchronous writes are not competing with a deep queue of non-synchronous writes.

Tuning this parameter is not expected to be effective for NVRAM-based storage arrays in the case where volumes are made of small number of spindles. However, when ZFS is presented with a volume made of a large (greater than 10) number of spindles, then this parameter can limit the read throughput obtained on the volume. The reason is that with a maximum of 10 or 35 queued I/Os per LUN, this can translate into less than 1 I/O per storage spindle, which is not enough for individual disks to deliver their IOPS. This issue would appear in iostat actv queue output approaching the value of zfs_vdev_max_pending.

Device drivers may also limit the number of outstanding I/Os per LUN. If you are using LUNs on storage arrays that can handle large numbers of concurrent IOPS, then the device driver constraints can limit concurrency. Consult the configuration for the drivers your system uses.

checking

# echo zfs_vdev_max_pending/D | mdb -k
zfs_vdev_max_pending:           10

setting dynamically

echo zfs_vdev_max_pending/W0t10 | mdb -kw

setting permanently

set zfs:zfs_vdev_max_pending=10