一、概序:
data abort 类型的KE比较常见,触发此KE的原因是,用户空间使用的地址都是虚拟地址,此地址经过MMU的负复杂
的页表映射到物理地址,当其中发生一些异常导致此虚拟地址无法访问到对应的物理地址时,就会通过报对应的BUG
使系统重启,此地址有可能已经被其他进程访问,也有可能因为部分硬件问题导致对应的地址出现翻转导致无法访问。
二、案例:
(1)硬件bitflip的KE:
堆栈信息如下:
[20512.223175] -(3)[30488:kworker/u8:2]Unable to handle kernel paging request at virtual address 4156106c
[20512.223201] -(3)[30488:kworker/u8:2]pgd = c0003000
[20512.223207] [4156106c] *pgd=80000040005003, *pmd=00000000
[20512.223223] -(3)[30488:kworker/u8:2]Internal error: Oops: 205 [#1] PREEMPT SMP ARM
[20512.223230] -(3)[30488:kworker/u8:2]Kernel Offset: disabled
[20513.223253] -(3)[30488:kworker/u8:2]PC is at set_task_cpu+0xd8/0x23c
[20513.223262] -(3)[30488:kworker/u8:2]LR is at walt_fixup_busy_time+0x1f0/0x4ac
[20513.223268] -(3)[30488:kworker/u8:2]pc : [<c02596f0>] lr : [<c028f46c>] psr: 60070093
使用GDB通过解析对应的符号表vmlinux可以看到堆栈如下:
(gdb) bt
#0 0xc02596f0 in set_task_rq (cpu=<optimized out>, p=<optimized out>)
at /home/buildsrv-108/jenkins/workspace/PY_UNIFIED_VERSION_BUILD/code/kernel-4.9/kernel/sched/sched.h:1061
#1 __set_task_cpu (cpu=<optimized out>, p=<optimized out>)
at /home/buildsrv-108/jenkins/workspace/PY_UNIFIED_VERSION_BUILD/code/kernel-4.9/kernel/sched/sched.h:1084
#2 set_task_cpu (p=0xdbcd4000, new_cpu=0) at /home/buildsrv-108/jenkins/workspace/PY_UNIFIED_VERSION_BUILD/code/kernel-4.9/kernel/sched/core.c:1314
#3 0xc025a648 in try_to_wake_up (p=0xdbcd4000, state=<optimized out>, wake_flags=<optimized out>)
at /home/buildsrv-108/jenkins/workspace/PY_UNIFIED_VERSION_BUILD/code/kernel-4.9/kernel/sched/core.c:2214
#4 0xc025a914 in wake_up_process (p=<optimized out>)
at /home/buildsrv-108/jenkins/workspace/PY_UNIFIED_VERSION_BUILD/code/kernel-4.9/kernel/sched/core.c:2294
#5 0xc0240bdc in wake_up_worker (pool=<optimized out>)
at /home/buildsrv-108/jenkins/workspace/PY_UNIFIED_VERSION_BUILD/code/kernel-4.9/kernel/workqueue.c:837
#6 process_one_work (worker=0xdbea5080, work=0xd5c5b434)
at /home/buildsrv-108/jenkins/workspace/PY_UNIFIED_VERSION_BUILD/code/kernel-4.9/kernel/workqueue.c:2076
#7 0xc0241998 in worker_thread (__worker=0xdbea5080)
at /home/buildsrv-108/jenkins/workspace/PY_UNIFIED_VERSION_BUILD/code/kernel-4.9/kernel/workqueue.c:2225
对应的set_task_cpu代码如下:
void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
{
......
if (task_cpu(p) != new_cpu) {
if (p->sched_class->migrate_task_rq)
p->sched_class->migrate_task_rq(p);
p->se.nr_migrations++;
perf_event_task_migrate(p);
walt_fixup_busy_time(p, new_cpu);
}
__set_task_cpu(p, new_cpu);
}
对应帧的反汇编代码如下:
(gdb) f 3
#3 0xc025a648 in try_to_wake_up (p=0xdbcd4000, state=<optimized out>, wake_flags=<optimized out>)
at /home/buildsrv-108/jenkins/workspace/PY_UNIFIED_VERSION_BUILD/code/kernel-4.9/kernel/sched/core.c:2214
2214 in /home/buildsrv-108/jenkins/workspace/PY_UNIFIED_VERSION_BUILD/code/kernel-4.9/kernel/sched/core.c
(gdb) i reg
r0 0xdbcd4080 3687661696
r1 0xdf79c900 3749300480
r2 0x4094d 264525
r3 0x4094d 264525
r4 0xdbcd4000 3687661568
r5 0xdbcd4644 3687663172
r6 0xc1404548 3242214728
gdb) disas
Dump of assembler code for function try_to_wake_up:
0xc025a62c <+552>: beq 0xc025a648 <try_to_wake_up+580>
0xc025a630 <+556>: ldr r3, [r11, #-56] ; 0x38
0xc025a634 <+560>: mov r1, r10
0xc025a638 <+564>: mov r0, r4 //将r4的值传给r0
0xc025a63c <+568>: orr r3, r3, #4
0xc025a640 <+572>: str r3, [r11, #-56] ; 0x38
0xc025a644 <+576>: bl 0xc0259618 <set_task_cpu> //跳转到set_task_cpu函数中
=> 0xc025a648 <+580>: movw r3, #17828 ; 0x45a4
从上面汇编代码可以看出r4的值应该和r0相等(也就是代码中p的值),但时间r0 的倒数第四位翻转为1,使访问的
地址发生变化:dbcd4000 ->dbcd4080,从此点可以看出是硬件Bitflip导致的KE,如果问题概率比较高的话,可
以通过交叉CPU/memory来验证此问题。
(2)踩内存触发的KE:
所谓踩内存,意思就是将要使用的这块内存已经其他地方非法占有,非法占有的方式有数组越界/use after free等,下面
看一个具体的实例,其中kernel log打印出来的堆栈信息如下:
[ 192.960966] (0)[1410:Signal Catcher]Unable to handle kernel paging request at virtual address 880646e1
[ 192.960998] (0)[1410:Signal Catcher]pgd = d06f4000
[ 192.961013] [880646e1] *pgd=00000000
[ 193.961221] -(0)[1410:Signal Catcher]PC is at find_vma+0x54/0x80
[ 193.961233] -(0)[1410:Signal Catcher]LR is at 0xd18ac3d8
通过GDB加载vmlinux解析出如下堆栈:
(gdb) bt
#0 find_vma (mm=0xdab73180, addr=3040309248) at /home/buildsrv-96/jenkins/workspace/UNIFIED_VERSION_BUILD-2/code/kernel-3.18/mm/mmap.c:2099
#1 0xc01171f8 in __do_page_fault (tsk=<optimized out>, flags=<optimized out>, fsr=<optimized out>, addr=<optimized out>, mm=<optimized out>)
at /home/buildsrv-96/jenkins/workspace/UNIFIED_VERSION_BUILD-2/code/kernel-3.18/arch/arm/mm/fault.c:232
#2 do_page_fault (addr=0, fsr=3040309248, regs=0xd0001fb0)
at /home/buildsrv-96/jenkins/workspace/UNIFIED_VERSION_BUILD-2/code/kernel-3.18/arch/arm/mm/fault.c:314
#3 0xc01003dc in do_DataAbort (addr=0, fsr=23, regs=0xd0001fb0)
看到第0帧的addr = 3040309248就可以明显发现很奇怪,一般不会出现这种异常的addr,下面接着分析,
(gdb) f 2
#2 do_page_fault (addr=0, fsr=3040309248, regs=0xd0001fb0)
at /home/buildsrv-96/jenkins/workspace/UNIFIED_VERSION_BUILD-2/code/kernel-3.18/arch/arm/mm/fault.c:314
314 in /home/buildsrv-96/jenkins/workspace/UNIFIED_VERSION_BUILD-2/code/kernel-3.18/arch/arm/mm/fault.c
切到第二帧的时候,可以看到addr = 0,并且在函数的传递过程中,addr的值并没有发生变化,这里可以看出addr
有被踩的可能,下面看汇编代码也可以很明显的看出addr被踩:
(gdb) disas
Dump of assembler code for function do_page_fault:
0xc0117130 <+0>: mov r12, sp
0xc0117134 <+4>: push {r4, r5, r6, r7, r8, r9, r10, r11, r12, lr, pc}
0xc0117138 <+8>: sub r11, r12, #4
0xc01171f0 <+192>: mov r0, r5
0xc01171f4 <+196>: bl 0xc0222e1c <find_vma>
=> 0xc01171f8 <+200>: subs r9, r0, #0 //r9 = r0 - 0=0
0xc01171fc <+204>: beq 0xc01173b8 <do_page_fault+648>
0xc0117200 <+208>: ldr r3, [r9]
0xc0117204 <+212>: cmp r8, r3
0xc0117208 <+216>: bcc 0xc0117390 <do_page_fault+608>
(gdb) i reg
r0 0x0 0
r1 0xb5377000 3040309248
r2 0xff000b2c 4278192940
r3 0x880646ed 2282112749
r4 0xd0001fb0 3489669040
r5 0xdab73180 3669438848
r6 0xd18ac100 3515531520
r7 0x17 23
r8 0xb5377000 3040309248
r9 0xb5377000 3040309248
r10 0xdab731b8 3669438904
上面汇编代码中r9中的值应该为0,但栈打印出来的是0xb5377000 = 3040309248,怀疑这个地址被踩了导致出现
的问题。对于踩内存的问题,需要打开slub或者kasan的debug机制来调试此类问题,当出现踩内存时可以将对应踩的
位置表示出来,具体方法可以参考博客:内存管理三 内核内存检测KASAN。
作者:frank_zyp
您的支持是对博主最大的鼓励,感谢您的认真阅读。
本文无所谓版权,欢迎转载。