My environment is write heavy. I have hundreds of millions of phones sending me events continously. They are like billions and billions of star like bubbles invading my spinning rust.
Consequently, when a disk fails or takes on excessive CRC errors a resilver job is kicked off. Similarly, I scrub the storage pools monthly and occasionally turn up a bad disk that is silently corrupting. The problem is the resilver or scrub job reports that it needs two or three months to complete, with the default settings. That doesn’t work for me. If you are reading this, you might have a similar problem.
I would prefer to throttle the inbound data and repair the system to optimal data redundancy. Around here, data integrity is everything. To complete the scrubs and resilvers quickly, I have had to tweak ZFS in an Evil way. First, we’ll talk about what to put into /etc/system, then I will show you how to make these changes take effect immediately.
The /etc/system changes are straight forward:
* Prioritize resilvering by setting the delay to zero
set zfs:zfs_resilver_delay = 0
* Prioritize scrubs by setting the delay to zero
set zfs:zfs_scrub_delay = 0
* set maximum number of inflight IOs to a reasonable value - this number will vary for your environment
set zfs:zfs_top_maxinflight = 128
* resilver for five seconds per TXG
set zfs:zfs_resilver_min_time_ms = 5000
The default value here is three seconds. By giving resilver five seconds per transaction group we can rebuild the mirror more quickly.
Now, to make these changes take effect immediately so you dont have to reboot, you can execute the following:
echo zfs_resilver_delay/w0 | mdb -kw
echo zfs_scrub_delay/w0 |mdb -kw
echo zfs_top_maxinflight/w7f |mdb -kw
echo zfs_resilver_min_time_ms/w1388 |mdb -kw
These variables and more can be found in dsl_scan.c
These modifications speed up my scrub/resilver operations from taking a approximately two months, to approximately two to three hours depending on the individual system.
Happy Scrubbing….
1711
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
