配置:
cat /proc/sys/kernel/hung_task_panic 软中断 , 内核中有进程进入了死循环,结束不了,或执行时间过长。
cat /proc/sys/kernel/nmi_watchdog 硬中断
PANIC: "Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2"
提示有hard LOCKUP
栈信息如下:
PID: 16104 TASK: ffff880118b8e080 CPU: 2 COMMAND: "bwc_shrinker"
#0 [ffff88002c287b50] machine_kexec at ffffffff81038fa9
#1 [ffff88002c287bb0] crash_kexec at ffffffff810c5992
#2 [ffff88002c287c80] panic at ffffffff81511980
#3 [ffff88002c287d00] watchdog_overflow_callback at ffffffff810e646d
#4 [ffff88002c287d20] __perf_event_overflow at ffffffff8111c357
#5 [ffff88002c287da0] perf_event_overflow at ffffffff8111c924
#6 [ffff88002c287db0] intel_pmu_handle_irq at ffffffff81022e07
#7 [ffff88002c287e90] perf_event_nmi_handler at ffffffff815161a9
#8 [ffff88002c287ea0] notifier_call_chain at ffffffff81517c65
#9 [ffff88002c287ee0] atomic_notifier_call_chain at ffffffff81517cca
#10 [ffff88002c287ef0] notify_die at ffffffff810a12ae
#11 [ffff88002c287f20] do_nmi at ffffffff8151595b
#12 [ffff88002c287f50] nmi at ffffffff81515220
[exception RIP: __read_lock_failed+8]
RIP: ffffffff8128c6a8 RSP: ffff880107f11c08 RFLAGS: 00000097
RAX: 0000000000000297 RBX: 0000000000000077 RCX: 0000000000000082
RDX: 0000000000000297 RSI: 0000000000000202 RDI: ffff88020b7633f4
RBP: ffff880107f11c10 R8: 0000000000000000 R9: 0000000000000000
R10: 00000000ffffffff R11: ffff88006e94a268 R12: ffffffffa0952840
R13: 0000000000000008 R14: ffff88006bd677c8 R15: 0000000000000202
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#13 [ffff880107f11c08] __read_lock_failed at ffffffff8128c6a8
#14 [ffff880107f11c08] _read_lock_irqsave at ffffffff815149f5
#15 [ffff880107f11c18] combo_clean_blocks_compress at ffffffffa0938126 [bwc]
#16 [ffff880107f11cb8] combo_blocks_compress at ffffffffa093899e [bwc]
#17 [ffff880107f11d58] pool_shrink at ffffffffa09344b0 [bwc]
#18 [ffff880107f11db8] sys_shrink at ffffffffa091b245 [bwc]
#19 [ffff880107f11e18] shrink_helper_main at ffffffffa091b2da [bwc]
#20 [ffff880107f11e38] worker_thread at ffffffff81094b00
#21 [ffff880107f11ee8] kthread at ffffffff8109acd6
#22 [ffff880107f11f48] kernel_thread at ffffffff8100c20a
==========================================================
soft lock_up
之前的panic信息被删除了,栈的大概意思是被NMI watchdog给中断了。
==========================================================
此类问题分析过程:
首先应该确定是hard lock_up还是soft lock_up
----------------------------------------------------------------------------------------------------------
hard lock_up是因为中断被禁掉了,长时间(默认应该是5S)没有打开,这时NMI(Non-maskable interrupt,不可屏蔽中断,简称NMI)
会中断当前进程,这时就要分析,当前进程为什么长时间关闭中断,目前我遇到一种情况:
1. 其他进程拿了锁,忘记放锁,另一个线程调用spin_lock_irqsave(),这个函数是先关闭中断,再去申请锁,如果一直拿不到锁,就会触发NMI。
这种问题的分析过程是这样的,在crash工具中执行foreach bt,把所有的进程栈都打印出来,看是否有进程在锁内的情况,如果有,分析为什么长时间没有放锁,
如果没有进程在锁内,则需要遍历代码,查找所有使用该锁的地方,看是否有忘记放锁的流程。
-------------------------------------------------------------------------------------------------------------
soft lock_up是由于抢占被长时间关闭,系统无法正常调度其他进程。这时NMI watchdog会中断该线程。
这种问题就需要查看代码,看为什么长时间死循环,可能是循环条件一直为真。