linux---"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message

 

2020年3月14日服务器异常重启，查看/var/log/message日志，如下：

Mar 14 15:04:59 localhost kernel: azx_single_send_cmd: 11 callbacks suppressed
Mar 14 15:05:14 localhost kernel: azx_single_send_cmd: 4 callbacks suppressed
Mar 14 15:07:14 localhost kernel: INFO: task jbd2/dm-1-8:1362 blocked for more than 120 seconds.
Mar 14 15:08:15 localhost kernel:      Tainted: G           OE     4.17.19-debug.x86_64 #1
Mar 14 15:08:32 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 14 15:08:43 localhost kernel: jbd2/dm-1-8     D    0  1362      2 0x80000000
Mar 14 15:08:43 localhost kernel: Call Trace:
Mar 14 15:08:43 localhost kernel: ? __schedule+0x28a/0x840
Mar 14 15:08:43 localhost kernel: ? __wake_up_common_lock+0x87/0xc0

Mar 14 15:08:43 localhost kernel: schedule+0x32/0x80
Mar 14 15:08:43 localhost kernel: jbd2_journal_commit_transaction+0x275/0x1a00 [jbd2]
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x40/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x34/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x34/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x40/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x34/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x40/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x34/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x40/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x34/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x34/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x40/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x34/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x34/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x40/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to_asm+0x34/0x70
Mar 14 15:08:43 localhost kernel: ? __switch_to+0xa8/0x480
Mar 14 15:08:43 localhost kernel: ? remove_wait_queue+0x60/0x60
Mar 14 15:08:43 localhost kernel: ? try_to_del_timer_sync+0x4d/0x80
Mar 14 15:08:43 localhost kernel: kjournald2+0xc1/0x260 [jbd2]
Mar 14 15:08:43 localhost kernel: ? remove_wait_queue+0x60/0x60
Mar 14 15:08:43 localhost kernel: kthread+0xf5/0x130
Mar 14 15:08:43 localhost kernel: ? commit_timeout+0x10/0x10 [jbd2]
Mar 14 15:08:43 localhost kernel: ? kthread_bind+0x10/0x10
Mar 14 15:08:43 localhost kernel: ret_from_fork+0x35/0x40

 

原因在于，至少情况下，Linux会使用最多40％的可用内存作为文件系统缓存。 将缓存写入磁盘时，有一个至少120秒的超时时间。出现上述问题的原因是IO的处理速度不够快，不能在120秒将缓存中的数据全部写入磁盘。IO系统响应缓慢，导致越来越多的请求堆积，最终系统内存全部被占用，导致系统失去响应。这是Linux延迟写入机制带来的问题，并且在主机内存重叠时，出现该问题的可能性增大。

我们可以从以下思路进行调优：

1. 减少脏数据的比例，避免刷写超时
2. 初步脏数据在内存中的存放时间，避免积少成多

修改相应参数

临时修改

# sysctl -w vm.dirty_ratio=10
# sysctl -w vm.dirty_background_ratio=5
# sysctl -p

永久修改

#vi /etc/sysctl.conf
写入
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

修改完重启服务器