一次PMC的RAID卡使用arcconf配置，写入性能极低的问题排查

问题现象

	将raid卡从LSI9460切换到PMC8204，安装系统后，测试发现写入数据库极其缓慢。
	使用fio测试发现写入性能只能达到30MB/s，与当前配置的3个HDD硬盘应该表现的实力相差很大。

排查和解决过程

3个HDD从LSI9460上取下来的，磁盘性能经过验证的，排除硬盘问题，唯一区别是raid卡，暂定判断是PMC这个raid卡的问题。
有2个思路：
1、raid卡故障
2、raid卡配置问题

1. raid卡故障排查
   raid卡制作、删除等都正常，制作多次都正常运行，只有写入很慢，似乎看起来并不是raid卡故障，带着疑问进行raid卡配置检查。
2. raid卡配置问题排查
   raid性能低的情况下，想起来其他raid卡的awb模式，查看pmc关于缓存的配置：

]# ./arcconf getconfig 1 ad

Controllers found: 1
----------------------------------------------------------------------
Controller information
----------------------------------------------------------------------
   Controller Status                        : Optimal
   Controller Mode                          : RAID (Hide RAW)
   Channel description                      : SCSI
   Controller Model                         : INSPUR Adaptec PM8204-2GB
   Controller Serial Number                 : RANB16EZ1468B30
   Controller World Wide Name               : 56C92BF006660F1E
   Physical Slot                            : 5
   Host bus type                            : PCIe 3.0
   Host bus speed                           : 7880 MBps
   Host bus link width                      : 8 bit(s)/link(s)
   PCI Address (Bus:Device:Function)        : 0:10:0:0
   Number of Ports                          : 2
   Internal Port Count                      : 2
   External Port Count                      : 0
   Defunct disk drive count                 : 0
   NCQ status                               : Enabled
   Queue Depth                              : Automatic
   Monitor and Performance Delay            : 60 minutes
   Elevator Sort                            : Enabled
   Degraded Mode Performance Optimization   : Disabled
   Latency                                  : Disabled
   Statistics data collection mode          : Disabled
   Post Prompt Timeout                      : 15 seconds
   Boot Controller                          : False
   Primary Boot Volume                      : None
   Secondary Boot Volume                    : None
   Driver Name                              : smartpqi
   Driver Supports SSD I/O Bypass           : Yes
   Manufacturing Part Number                : Not Applicable
   Manufacturing Spare Part Number          : Not Applicable
   Manufacturing Wellness Log               : Not Applicable
   NVRAM Checksum Status                    : Passed
   Sanitize Lock Setting                    : None
   --------------------------------------------------------
   Power Settings
   --------------------------------------------------------
   Power Consumption                        : Not Available
   Current Power Mode                       : Maximum Performance
   Pending Power Mode                       : Not Applicable
   Survival Mode                            : Disabled
   --------------------------------------------------------
   I2C Settings
   --------------------------------------------------------
   I2C Address                              : 0xDE
   I2C Clock Speed                          : 400 KHZ
   I2C Clock Stretching                     : Enabled
   --------------------------------------------------------
   Cache Properties
   --------------------------------------------------------
   Cache Status                             : Ok
   Cache State                              : Error Detected Run Adu
   Cache Serial Number                      : Not Applicable
   Cache memory                             : 1808 MB
   Read Cache Percentage                    : 100 percent
   Write Cache Percentage                   : 0 percent
   No-Battery Write Cache                   : Disabled
   Wait for Cache Room                      : Disabled
   Write Cache Bypass Threshold Size        : 1040 KB
   --------------------------------------------------------
   Green Backup Information
   --------------------------------------------------------
   Backup Power Status                      : No Battery Connected
   Battery/Capacitor Pack Count             : 0
   Hardware Error                           : No Error
   --------------------------------------------------------
   Physical Drive Write Cache Policy Information
   --------------------------------------------------------
   Configured Drives                        : Default
   Unconfigured Drives                      : Default
   HBA Drives                               : Default
   --------------------------------------------------------
   maxCache Properties
   --------------------------------------------------------
   maxCache Version                         : 4
   maxCache RAID5 WriteBack Enabled         : Enabled
   --------------------------------------------------------
   RAID Properties
   --------------------------------------------------------
   Logical devices/Failed/Degraded          : 2/0/0
   Spare Activation Mode                    : Disabled
   Background consistency check             : Idle
   Consistency Check Delay                  : 3 seconds
   Parallel Consistency Check Supported     : Enabled
   Parallel Consistency Check Count         : 1
   Inconsistency Repair Policy              : Disabled
   Consistency Check Inconsistency Notify   : Disabled
   Rebuild Priority                         : High
   Expand Priority                          : Medium
   --------------------------------------------------------
   Controller Version Information
   --------------------------------------------------------
   Firmware                                 : 5.32[0]
   Driver                                   : Linux 1.1.4-130
   Hardware Revision                        : B
   --------------------------------------------------------
   Connector information
                   
   --------------------------------------------------------
   Connector #0                             
      Connector Name                        : CN0
      Functional Mode                       : RAID (Hide RAW)
      Connector Location                    : Internal
      SAS Address                           : 56C92BF006660F1E


   Connector #1                             
      Connector Name                        : CN1
      Functional Mode                       : RAID (Hide RAW)
      Connector Location                    : Internal
      SAS Address                           : 56C92BF006660F22




Command completed successfully.

发现上面配置中有部分是关于读写的，发现Write Cache Percentage是0。

   Cache Properties
   --------------------------------------------------------
   Cache Status                             : Ok
   Cache State                              : Error Detected Run Adu
   Cache Serial Number                      : Not Applicable
   Cache memory                             : 1808 MB
   Read Cache Percentage                    : 100 percent
   Write Cache Percentage                   : 0 percent
   No-Battery Write Cache                   : Disabled
   Wait for Cache Room                      : Disabled
   Write Cache Bypass Threshold Size        : 1040 KB

查看帮助，尝试修改

]# ./arcconf setcache 1 
Controllers found: 1
Invalid arguments.

 Usage: SETCACHE <Controller#> LOGICALDRIVE <LogicalDrive# | All> <logical mode> [noprompt] [nologs]
 Usage: SETCACHE <Controller#> DRIVEWRITECACHEPOLICY <drive Type> <drive cache policy> [noprompt] [nologs]
 Usage: SETCACHE <Controller#> CACHERATIO <read#> <write#>
 Usage: SETCACHE <Controller#> WAITFORCACHEROOM <enable | disable>
 Usage: SETCACHE <Controller#> NOBATTERYWRITECACHE <enable | disable>
 Usage: SETCACHE <Controller#> WRITECACHEBYPASSTHRESHOLD <threshold size>
 Usage: SETCACHE <Controller#> RECOVERCACHEMODULE
 Example: SETCACHE 1 LOGICALDRIVE 0 con 
 Example: SETCACHE 1 LOGICALDRIVE All con 
 Example: SETCACHE 1 DRIVEWRITECACHEPOLICY Configured  0 Unconfigured 1 hba 2
 Example: SETCACHE 1 CACHERATIO 60 40 
 Example: SETCACHE 1 WAITFORCACHEROOM enable 
 Example: SETCACHE 1 NOBATTERYWRITECACHE enable 
 Example: SETCACHE 1 WRITECACHEBYPASSTHRESHOLD 1040 
 Example: SETCACHE 1 RECOVERCACHEMODULE

按照设置应该是设置CACHERATIO参数

]# ./arcconf setcache 1 cacheratio 10 90
Controllers found: 1
The specified operation cannot be performed as battery status is not Ok and No Battery Write Cache is disabled. Please ensure battery unit is fully charged or No Battery Write Cache is enabled.

Command aborted.

按当前的提示是PMC的raid卡没有带电池，无法设置，需要设置 No Battery Write Cache is enabled

]# ./arcconf setcache 1 NOBATTERYWRITECACHE enable
Controllers found: 1
WARNING : Enabling write caching on a cache module without a fully charged battery or capacitor could cause data loss in the event of a power failure.

Are you sure you want to continue?
Press y, then ENTER to continue or press ENTER to abort: y


Command completed successfully.

设置完成后，再设置读写

]# ./arcconf setcache 1 cacheratio 10 90
Controllers found: 1
WARNING : Enabling write caching on a cache module without a fully charged battery or capacitor could cause data loss in the event of a power failure.

Are you sure you want to continue?
Press y, then ENTER to continue or press ENTER to abort: y


Command completed successfully.

查看设置是否生效

]# ./arcconf getconfig 1 ad | grep -i cache
   Cache Properties
   Cache Status                               : Ok
   Cache Serial Number                        : Not Applicable
   Cache memory                               : 1808 MB
   Read Cache Percentage                      : 10 percent
   Write Cache Percentage                     : 90 percent
   No-Battery Write Cache                     : Enabled
   Wait for Cache Room                        : Disabled
   Write Cache Bypass Threshold Size          : 1040 KB
   Physical Drive Write Cache Policy Information
   maxCache Properties
   maxCache Version                           : 4
   maxCache RAID5 WriteBack Enabled           : Enabled

发现已经生效，进行fio测试

]# fio -filename=/opt/data1/lh -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=100M -size=2G -numjobs=1 -runtime=60 -group_reporting -name=mytest
mytest: (g=0): rw=write, bs=(R) 100MiB-100MiB, (W) 100MiB-100MiB, (T) 100MiB-100MiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 thread
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=300MiB/s][r=0,w=3 IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=1): err= 0: pid=20582: Mon May 15 17:53:00 2023
  write: IOPS=4, BW=467MiB/s (490MB/s)(2000MiB/4280msec)
    clat (msec): min=18, max=1674, avg=195.95, stdev=405.77
     lat (msec): min=31, max=1695, avg=213.74, stdev=405.95
    clat percentiles (msec):
     |  1.00th=[   18],  5.00th=[   18], 10.00th=[   19], 20.00th=[   20],
     | 30.00th=[   21], 40.00th=[   21], 50.00th=[   22], 60.00th=[   47],
     | 70.00th=[  103], 80.00th=[  211], 90.00th=[  236], 95.00th=[  936],
     | 99.00th=[ 1670], 99.50th=[ 1670], 99.90th=[ 1670], 99.95th=[ 1670],
     | 99.99th=[ 1670]
   bw (  KiB/s): min=204800, max=2252800, per=100.00%, avg=737116.40, stdev=864129.77, samples=5
   iops        : min=    2, max=   22, avg= 7.00, stdev= 8.54, samples=5
  lat (msec)   : 20=20.00%, 50=40.00%, 100=5.00%, 250=25.00%, 1000=5.00%
  cpu          : usr=8.30%, sys=0.56%, ctx=25, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,20,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=467MiB/s (490MB/s), 467MiB/s-467MiB/s (490MB/s-490MB/s), io=2000MiB (2097MB), run=4280-4280msec

Disk stats (read/write):
  sdc: ios=197/5911, merge=0/58, ticks=7203/435734, in_queue=423620, util=100.00%

这下大功告成，确实是pmc的raid卡配置问题，而不是raid卡的问题。

不过这个配置倒是有点不太理解为啥要做成这样，罢了罢了。