jbd2导致系统IO使用率高问题

收到IO占用高告警

系统信息：Linux version 2.6.32-696.18.7.1.el6.ucloud.x86_64 (root@59c188f3c79d) (gcc version 4.4.6 20120305  
(Red Hat 4.4.6-5) (GCC) ) #1 SMP Fri Jan 5 16:48:58 CST 2018 

1、到机器上看到io使用率忽高，同时iowait也高

$ iostat -d -x 1
Linux 2.6.32-696.18.7.1.el6.ucloud.x86_64 (bd-prod-acc-web0)     07/08/2019     _x86_64_    (2 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     1.17    0.00    0.88     0.06    16.40    18.58     0.01   16.88    0.90   16.92   4.06   0.36
vdb               0.00     0.86    0.00    1.03     0.00    15.14    14.68     0.01   11.48    1.79   11.48   4.56   0.47

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.00    0.00    1.00     0.00    72.00    72.00     1.00 2387.00    0.00 2387.00 1000.00 100.00
vdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     2.06    0.00    0.00    0.00   0.00 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     1.00    0.00    0.00    0.00   0.00 100.00
vdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     3.00    0.00    0.00    0.00   0.00 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.00    0.00    1.00     0.00     8.00     8.00     0.51 2258.00    0.00 2258.00 513.00  51.30
vdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     3.00    0.00    0.00    0.00   0.00 100.00

2、找到占用高的进程

$ sudo iotop 
Total DISK READ: 0.00 B/s | Total DISK WRITE: 51.21 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
  810 be/3 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % [jbd2/vdb-8] #这里看到是系统进程
 2614 be/4 root        0.00 B/s    0.00 B/s  0.00 % 99.99 % sadc -F -L -S DISK -S DISK 1 1 -
  412 be/3 root        0.00 B/s    3.94 K/s  0.00 % 99.99 % [jbd2/vda1-8]

3、jdb2进程占用IO资源较高，jdb2进程是文件系统ext4的写日志进程，占用高是内核一个bug

4、（1）升级内核
$ sudo yum update kernel 
或
(2)通过修改文件系统的提交次数来降低
 commit=60

或（3）重启 

Wait IO problem can be caused by several reasons, the basic road-map to find out is which process is "eating" your CPU first and then determine why. The main cause are those background processes with "D" status code which means "Uninterruptiable sleep". But the those processes with "D+" which means "Uninterruptible sleep foreground process" will generally not cause the serious problem as those background processes. In this example, the cause of Wait IO is the File System Journal, so the configuration of file system is the cause of the problem. 

By Quentin Sherman Xue - CEO  (http://www.chileoffshore.com/en/interesting-articles/126-linux-wait-io-problem)

系统io问题的一般思路：

查看系统总体IO性能------->找到占用高的进程-------->查看进程对应的IO操作-------->结合场景分析，根据场景优化