民用SSD 关闭 写cache后的 糟糕性能表现

昨天刚到手的一块SSD，建兴 ZETA 256GB的硬盘。

smartctl查看到的信息如下：

=== START OF INFORMATION SECTION ===
Device Model:     LITEON LCH-256V2S
Serial Number:    002448118079
LU WWN Device Id: 5 002303 10035169e
Firmware Version: FC8020A
User Capacity:    256,060,514,304 bytes [256 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sun Dec 20 21:52:23 2015 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, frozen [SEC2]
Write SCT (Get) XXX Error Recovery Control Command failed: scsi error aborted command
Wt Cache Reorder: N/A

在打开写缓存时，pg_test_fsync测试到的fsync写性能如下：

postgres@digoal-> pg_test_fsync 
5 seconds per test
O_DIRECT supported on this platform for open_datasync and open_sync.

Compare file sync methods using one 8kB write:
(in wal_sync_method preference order, except fdatasync is Linux's default)
        open_datasync                     14640.971 ops/sec      68 usecs/op
        fdatasync                         13786.079 ops/sec      73 usecs/op
        fsync                              9384.938 ops/sec     107 usecs/op
        fsync_writethrough                            n/a
        open_sync                          9606.181 ops/sec     104 usecs/op

Compare file sync methods using two 8kB writes:
(in wal_sync_method preference order, except fdatasync is Linux's default)
        open_datasync                      8157.733 ops/sec     123 usecs/op
        fdatasync                         10784.381 ops/sec      93 usecs/op
        fsync                              7117.690 ops/sec     140 usecs/op
        fsync_writethrough                            n/a
        open_sync                          4913.559 ops/sec     204 usecs/op

Compare open_sync with different write sizes:
(This is designed to compare the cost of writing 16kB in different write
open_sync sizes.)
         1 * 16kB open_sync write          7361.125 ops/sec     136 usecs/op
         2 *  8kB open_sync writes         5274.986 ops/sec     190 usecs/op
         4 *  4kB open_sync writes         3099.050 ops/sec     323 usecs/op
         8 *  2kB open_sync writes         2048.264 ops/sec     488 usecs/op
        16 *  1kB open_sync writes         1225.877 ops/sec     816 usecs/op

Test if fsync on non-write file descriptor is honored:
(If the times are similar, fsync() can sync data written on a different
descriptor.)
        write, fsync, close                9403.945 ops/sec     106 usecs/op
        write, close, fsync                9847.864 ops/sec     102 usecs/op

Non-sync'ed 8kB writes:
        write                            337716.595 ops/sec       3 usecs/op

看起来非常漂亮的性能指标。居然比我以前用的PCI-E接口的ocz还要好，有点不敢相信。

当我把硬盘的写缓存关闭后，再次测试，真实性能暴露。

[root@digoal parameters]# hdparm -W 0 /dev/sda

/dev/sda:
 setting drive write-caching to 0 (off)
 write-caching =  0 (off)

性能结果让人咂舌：

postgres@digoal-> pg_test_fsync 
5 seconds per test
O_DIRECT supported on this platform for open_datasync and open_sync.

Compare file sync methods using one 8kB write:
(in wal_sync_method preference order, except fdatasync is Linux's default)
        open_datasync                       527.295 ops/sec    1896 usecs/op
        fdatasync                           519.180 ops/sec    1926 usecs/op
        fsync                               168.596 ops/sec    5931 usecs/op
        fsync_writethrough                            n/a
        open_sync                           169.740 ops/sec    5891 usecs/op

Compare file sync methods using two 8kB writes:
(in wal_sync_method preference order, except fdatasync is Linux's default)
        open_datasync                       261.551 ops/sec    3823 usecs/op
        fdatasync                           517.381 ops/sec    1933 usecs/op
        fsync                               168.195 ops/sec    5945 usecs/op
        fsync_writethrough                            n/a
        open_sync                            84.394 ops/sec   11849 usecs/op

Compare open_sync with different write sizes:
(This is designed to compare the cost of writing 16kB in different write
open_sync sizes.)
         1 * 16kB open_sync write           168.287 ops/sec    5942 usecs/op
         2 *  8kB open_sync writes           84.500 ops/sec   11834 usecs/op
         4 *  4kB open_sync writes           42.232 ops/sec   23678 usecs/op
         8 *  2kB open_sync writes           20.911 ops/sec   47823 usecs/op
        16 *  1kB open_sync writes           10.288 ops/sec   97204 usecs/op

Test if fsync on non-write file descriptor is honored:
(If the times are similar, fsync() can sync data written on a different
descriptor.)
        write, fsync, close                 170.290 ops/sec    5872 usecs/op
        write, close, fsync                 171.121 ops/sec    5844 usecs/op

Non-sync'ed 8kB writes:
        write                            332315.202 ops/sec       3 usecs/op

用dd加direct,sync测试持续写入，在开启了写缓存的时候，性能还是很不错的，连续写入8GB数据。

但是在写入过程中，使用smartctl -x /dev/sda查看硬盘的信息时，又hang了。iowait飙升到100，问题可能还和NCQ有关，见上一篇文章。

如果这是一个普遍现象，企业用的话，还是要企业级的产品。