crash信息
crash> bt |grep -n2 RIP
8- #6 [ffff880046ec7f40] do_nmi at ffffffff8139f6f8
9- #7 [ffff880046ec7f50] nmi at ffffffff8139ed8a
10: [exception RIP: _spin_lock_irq+0x16]
11: RIP: ffffffff8139e726 RSP: ffff8803fdfafb90 RFLAGS: 00000002
12- RAX: 0000000000000000 RBX: 000000000000afc8 RCX: 000000000000cf91
13- RDX: 0000000000000001 RSI: 000000000000afc8 RDI: ffff88086c788960
--
30-#19 [ffff8803fdfaff40] sys_ioctl at ffffffff8110f611
31-#20 [ffff8803fdfaff80] system_call_fastpath at ffffffff81002f7b
32: RIP: 00007f117286fbd7 RSP: 00007f116cf2ba60 RFLAGS: 00000246
33- RAX: 0000000000000010 RBX: ffffffff81002f7b RCX: 00000000059cafe0
34- RDX: 0000000000811670 RSI: 00000000c0484c14 RDI: 000000000000000b
crash>
基础知识
- _spin_lock_irq 函数是内核自旋锁的关键函数,上述exception RIP: _spin_lock_irq+0x16 指明当前内核正在此行运行
- RDI: ffff88086c788960 是_spin_lock_irq的第一个参数的地址,也就是spin_lock 结构体的地址
进一步解析
- 这个对应的spin_lock地址为ffff88086c788960
crash> struct spinlock_t 0xffff88086c788960
struct spinlock_t {
raw_lock = {
slock = 0x10001 <-竞争该锁的CPU个数为0,0x0001-0x0001=0
}
}
-
slock值非常奇怪,常见的spinlock死锁,这里的slock高4位与低4位会有一个大于1的差值。
但是这个spinlock并没有这样的差值,理论上就不应该一直循环取锁,而应该迅速获得锁之后开启中断,也就不会导致NMI watchdog超时了。 -
除CPU6以外有4个cpu卡在大内核锁中
crash> bt -a|grep "exception RIP"
[exception RIP: lock_kernel+0x2c]
[exception RIP: lock_kernel+0x2c]
[exception RIP: VHBA_InitUpCmdByBio+0x3e]
[exception RIP: lock_kernel+0x2a]
[exception RIP: _spin_lock_irq+0x16]
[exception RIP: lock_kernel+0x33]
[exception RIP: mwait_idle+0x62]
[exception RIP: mwait_idle+0x62]
[exception RIP: mwait_idle+0x62]
- 同时获取大内核锁的地址和数据为:
crash> p kernel_flag
kernel_flag = $4 = {
raw_lock = {
slock = 0xc122c11d
}
}
crash> eval 0xc122 - 0xc11d
hexadecimal: 5
decimal: 5 <-竞争大内核锁的进程总数
octal: 5
binary: 0000000000000000000000000000000000000000000000000000000000000101
- 也就是说,已经有一个进程获取了大内核锁没有释放,另外4个需要大内核锁的CPU就一直处于死锁状态了。