linux kernel oops

最新推荐文章于 2021-04-30 11:24:19 发布

jiayinjia1983116

最新推荐文章于 2021-04-30 11:24:19 发布

阅读量2.1k

点赞数

分类专栏： linux kernel及driver

linux kernel及driver 专栏收录该内容

13 篇文章 0 订阅

订阅专栏

如何快速的定位oops

一. addr2line 工具

Addr2line 工具（它是标准的 GNU Binutils 中的一部分）是一个可以将指令的地址和可执行映像转换成文件名、函数名和源代码行数的工具。这种功能对于将跟踪地址转换成更有意义的内容来说简直是太棒了。

有了 addr2line 工具，遇到oops时，我们就可以找到当前PC（EIP）所在地址，然后就可转换为相应的C文件里具体一行。接下来就是仔细分析附近代码，找出导致出错的代码，这个能力可能需要一定的经验积累。

以下为使用addr2line的例子，我直接从http://elinux.org/Addr2line_for_kernel_debugging copy下来。

Introduction

In this howto we will look how to debug a kernel without KGDB(for instance if we don't have serial available)

Howto

Here's an example of Oops(null pointer exception):

# while true;do dmesg -c;done
[   87.915496] request_suspend_state: sleep (0->3) at 82181848246 (2010-05-22 19:29:45.776824970 UTC)
[   88.028686] deinit sharp panel
[   88.635070] Unable to handle kernel NULL pointer dereference at virtual address 00000fc5
[   88.635101] pgd = c5db8000
[   88.635131] [00000fc5] *pgd=16aba031, *pte=00000000, *ppte=00000000
[   88.635162] Internal error: Oops: 11 [#1] PREEMPT
[   88.635192] last sysfs file: /sys/power/state
[   88.635192] Modules linked in:
[   88.635223] CPU: 0    Not tainted  (2.6.32 #385)
[   88.635253] PC is at msmfb_suspend+0x1c/0x2c
[   88.635284] LR is at msmfb_suspend+0x20/0x2c
[   88.635314] pc : [<c01b063c>]    lr : [<c01b0640>]    psr: a0000013
[   88.635314] sp : c6847f38  ip : c6847f50  fp : c6847f4c
[   88.635345] r10: c6847f84  r9 : c68009a8  r8 : c04e47e8
[   88.635345] r7 : c04e4818  r6 : c68e6ea0  r5 : c0535c1c  r4 : c68e6ea0
[   88.635375] r3 : 00000fc5  r2 : c04e1f64  r1 : c6846000  r0 : 00000fc5
[   88.635406] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[   88.635437] Control: 00c5387d  Table: 15db8008  DAC: 00000017
[   88.635437] 
[   88.635437] PC: 0xc01b05bc:
[   88.635467] 05bc  e58de000 e58dc008 e58dc004 ebfffe7d e24bd014 e89da830 e1a0c00d e92dd830
[   88.635528] 05dc  e24cb004 e24dd010 e1a04001 e1a05000 ebfff6df e594000c e894000e e082e000
[   88.635559] 05fc  e3a0c000 e1a00005 e0813003 e58de000 e58dc008 e58dc004 ebfffe6a e24bd014
[   88.635620] 061c  e89da830 e1a0c00d e92dd818 e24cb004 e5103038 e1a04000 e1a00003 e1a0e00f
[   88.635681] 063c  e593f000 e2840074 eb079177 e89da818 e1a0c00d e92ddff0 e24cb004 e24dd02c
[   88.635742] 065c  e5907004 e1a04000 e10f6000 f10c0080 e3a00001 ebfa5e63 e24b0034 ebfb087b
[   88.635772] 067c  e3a01e11 e51b2034 e18100d4 e51b3030 e0520000 e0c31001 e2502000 b28225ee
[   88.635833] 069c  b282296b e59f0290 b2822c0a e1a03fc2 e0e32091 e3530000 da00005b e59f027c
[   88.635894] 
[   88.635894] LR: 0xc01b05c0:
[   88.635894] 05c0  e58dc008 e58dc004 ebfffe7d e24bd014 e89da830 e1a0c00d e92dd830 e24cb004
[   88.635955] 05e0  e24dd010 e1a04001 e1a05000 ebfff6df e594000c e894000e e082e000 e3a0c000
[   88.636016] 0600  e1a00005 e0813003 e58de000 e58dc008 e58dc004 ebfffe6a e24bd014 e89da830
[   88.636077] 0620  e1a0c00d e92dd818 e24cb004 e5103038 e1a04000 e1a00003 e1a0e00f e593f000
[   88.636108] 0640  e2840074 eb079177 e89da818 e1a0c00d e92ddff0 e24cb004 e24dd02c e5907004
[   88.636169] 0660  e1a04000 e10f6000 f10c0080 e3a00001 ebfa5e63 e24b0034 ebfb087b e3a01e11
[   88.636230] 0680  e51b2034 e18100d4 e51b3030 e0520000 e0c31001 e2502000 b28225ee b282296b
[   88.636260] 06a0  e59f0290 b2822c0a e1a03fc2 e0e32091 e3530000 da00005b e59f027c e59f127c
[   88.636322] 
[   88.636322] SP: 0xc6847eb8:
[   88.636352] 7eb8  c6847f04 c6847ec8 c020a08c c0085e50 c69511a4 00000002 ffffffff c6847f24
[   88.636383] 7ed8  c68e6ea0 c04e4818 c6847f4c c6847ef0 c0028b4c c0028300 00000fc5 c6846000
[   88.636444] 7ef8  c04e1f64 00000fc5 c68e6ea0 c0535c1c c68e6ea0 c04e4818 c04e47e8 c68009a8
[   88.636505] 7f18  c6847f84 c6847f4c c6847f50 c6847f38 c01b0640 c01b063c a0000013 ffffffff
[   88.636566] 7f38  c01b0620 c6846000 c6847f74 c6847f50 c0083dc8 c01b062c c04e47ec c68009a0
[   88.636596] 7f58  c6846000 c68009a0 c0083cd0 00000000 c6847fc4 c6847f78 c00640f0 c0083cdc
[   88.636657] 7f78  00000000 c04e47ec 00000000 00000000 c6825160 c00684b8 c6847f90 c6847f90
[   88.636718] 7f98  c6847fc4 c6829ee0 00000002 c0063fa4 c68009a0 00000000 00000000 00000000
[   88.636749] 
[   88.636779] IP: 0xc6847ed0:
[   88.636779] 7ed0  ffffffff c6847f24 c68e6ea0 c04e4818 c6847f4c c6847ef0 c0028b4c c0028300
[   88.636840] 7ef0  00000fc5 c6846000 c04e1f64 00000fc5 c68e6ea0 c0535c1c c68e6ea0 c04e4818
[   88.636871] 7f10  c04e47e8 c68009a8 c6847f84 c6847f4c c6847f50 c6847f38 c01b0640 c01b063c
[   88.636932] 7f30  a0000013 ffffffff c01b0620 c6846000 c6847f74 c6847f50 c0083dc8 c01b062c
[   88.636993] 7f50  c04e47ec c68009a0 c6846000 c68009a0 c0083cd0 00000000 c6847fc4 c6847f78
[   88.637023] 7f70  c00640f0 c0083cdc 00000000 c04e47ec 00000000 00000000 c6825160 c00684b8
[   88.637084] 7f90  c6847f90 c6847f90 c6847fc4 c6829ee0 00000002 c0063fa4 c68009a0 00000000
[   88.637145] 7fb0  00000000 00000000 c6847ff4 c6847fc8 c0068260 c0063fb0 00000000 00000000
[   88.637207] 
[   88.637207] FP: 0xc6847ecc:
[   88.637207] 7ecc  00000002 ffffffff c6847f24 c68e6ea0 c04e4818 c6847f4c c6847ef0 c0028b4c
[   88.637268] 7eec  c0028300 00000fc5 c6846000 c04e1f64 00000fc5 c68e6ea0 c0535c1c c68e6ea0
[   88.637298] 7f0c  c04e4818 c04e47e8 c68009a8 c6847f84 c6847f4c c6847f50 c6847f38 c01b0640
[   88.637359] 7f2c  c01b063c a0000013 ffffffff c01b0620 c6846000 c6847f74 c6847f50 c0083dc8
[   88.637420] 7f4c  c01b062c c04e47ec c68009a0 c6846000 c68009a0 c0083cd0 00000000 c6847fc4
[   88.637481] 7f6c  c6847f78 c00640f0 c0083cdc 00000000 c04e47ec 00000000 00000000 c6825160
[   88.637512] 7f8c  c00684b8 c6847f90 c6847f90 c6847fc4 c6829ee0 00000002 c0063fa4 c68009a0
[   88.637573] 7fac  00000000 00000000 00000000 c6847ff4 c6847fc8 c0068260 c0063fb0 00000000
[   88.637634] 
[   88.637634] R1: 0xc6845f80:
[   88.637634] 5f80  00000000 00000000 c68245c0 c004b5bc c04e4220 c04e4220 c6845fc4 c6829f40
[   88.637695] 5fa0  00000002 c006f2f4 00000000 00000000 00000000 00000000 c6845ff4 c6845fc8
[   88.637756] 5fc0  c0068260 c006f300 00000000 00000000 c6845fd0 c6845fd0 00000000 00000000
[   88.637786] 5fe0  00000000 00000000 00000000 c6845ff8 c00299d8 c00681e4 ffffffff ffffffff
[   88.637847] 6000  00000000 00000002 00000000 c6825160 c04e2530 00000000 00000017 c6825160
[   88.637908] 6020  c04e2060 c6846000 c04e2060 c6b58900 c6b58900 c6a08ba0 c6847d24 c6847cc0
[   88.637939] 6040  c0393f34 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   88.638000] 6060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   88.638061] 
[   88.638061] R2: 0xc04e1ee4:
[   88.638061] 1ee4  00000090 00000000 c00417f8 c0435ac4 c0435acc c0435ad0 00000000 00000002
[   88.638122] 1f04  00000003 00000005 0000001d 00000004 c0435954 c000e478 00000000 00000000
[   88.638153] 1f24  00000000 00000000 c0434ed0 c04f9820 00000000 00000000 00000000 c01ddfcc
[   88.638214] 1f44  00000000 00000000 00000000 00000000 00000000 c04e1f7c c68f81e0 00000000
[   88.638275] 1f64  c04e4818 c68e6ea0 00000097 c00424a0 c00423f0 00000001 00000000 00000000
[   88.638305] 1f84  00000000 00000000 00000000 00000000 00000000 00000000 c0042454 c004240c
[   88.638366] 1fa4  00000000 00000000 00000000 00000000 00000000 00000000 00000000 c0042c94
[   88.638427] 1fc4  c0042be4 00000000 00000000 00000000 c0434ecc c04f9820 00000000 00000000
[   88.638458] 
[   88.638458] R4: 0xc68e6e20:
[   88.638488] 6e20  c68e0910 c692c180 00000001 c7200000 00000000 c053bd2c 00000000 00000000
[   88.638549] 6e40  c68e6e50 00000000 00000000 00000000 c68e6c00 c68e09c8 00000140 000001e0
[   88.638580] 6e60  00000000 00000000 00000fc5 00000fc4 00000004 00000650 00000000 00000000
[   88.638641] 6e80  00000140 000001e0 c680a000 c6973eec c04fa208 00000032 c01b0b28 00000000
[   88.638702] 6ea0  c04e1f64 c6951188 00000096 c01b0620 c01b1128 00000000 c5d95b58 c053bea0
[   88.638732] 6ec0  00000101 c0450d78 ffffab02 00000000 00000973 00000000 00000000 00000000
[   88.638793] 6ee0  1fc8f477 00000015 00000000 00000000 0f8c01d0 00000000 0b248caa 00000052
[   88.638854] 6f00  00000001 c68e6f04 c68e6f04 c68e6f0c c68e6f0c c6991f20 c6991f40 c68e6f1c
[   88.638885] 
[   88.638885] R5: 0xc0535b9c:
[   88.638916] 5b9c  00000000 00000000 00000000 00000000 00000000 c05408b8 c0512988 00000300
[   88.638977] 5bbc  c043bf90 7fffffff 00000000 00000000 00000000 00000000 00000000 00000000
[   88.639007] 5bdc  00000000 00000000 00000000 00000000 00000000 0d1cef11 00000000 0d1cef11
[   88.639068] 5bfc  00000000 c05425a0 c050fcd8 c695e348 c695e348 00000000 000001b1 00000000
[   88.639099] 5c1c  00000003 00000000 00000000 00000000 00000000 c68d5d00 00000114 00000000
[   88.639160] 5c3c  00000000 00000000 00000000 00000002 00000000 0000253b 00000000 00000000
[   88.639221] 5c5c  00000000 00000000 00000000 00000000 00000000 00000000 00001d9f 00000000
[   88.639251] 5c7c  00000000 0000090d 00000000 0000125e 00000000 00000003 00000000 00000000
[   88.639312] 
[   88.639312] R6: 0xc68e6e20:
[   88.639343] 6e20  c68e0910 c692c180 00000001 c7200000 00000000 c053bd2c 00000000 00000000
[   88.639373] 6e40  c68e6e50 00000000 00000000 00000000 c68e6c00 c68e09c8 00000140 000001e0
[   88.639434] 6e60  00000000 00000000 00000fc5 00000fc4 00000004 00000650 00000000 00000000
[   88.639495] 6e80  00000140 000001e0 c680a000 c6973eec c04fa208 00000032 c01b0b28 00000000
[   88.639526] 6ea0  c04e1f64 c6951188 00000096 c01b0620 c01b1128 00000000 c5d95b58 c053bea0
[   88.639587] 6ec0  00000101 c0450d78 ffffab02 00000000 00000973 00000000 00000000 00000000
[   88.639648] 6ee0  1fc8f477 00000015 00000000 00000000 0f8c01d0 00000000 0b248caa 00000052
[   88.639678] 6f00  00000001 c68e6f04 c68e6f04 c68e6f0c c68e6f0c c6991f20 c6991f40 c68e6f1c
[   88.639739] 
[   88.639739] R7: 0xc04e4798:
[   88.639739] 4798  00000000 00000000 00000000 00000000 00000000 00000000 00000000 c00828f8
[   88.639801] 47b8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   88.639862] 47d8  00000001 00000001 c04e47e0 c04e47e0 00000001 c68009a0 c04e47f0 c04e47f0
[   88.639892] 47f8  c0083cd0 00000000 c04e4800 c04e4800 c0083abc 00000000 c04e4810 c04e4810
[   88.639953] 4818  c050033c c04e1f64 c0083eb0 c04e483c 00000000 0000003c 00000043 c68222e0
[   88.640014] 4838  00000000 c0084170 00000000 00000000 00000078 00008000 0000000a 00000000
[   88.640045] 4858  00000000 c0535c34 c0086a94 c04dd474 00000000 00000000 00000000 c68bd6c0
[   88.640106] 4878  00120001 00000000 00000001 00000114 00000000 00000000 00000000 c04e4894
[   88.640167] 
[   88.640167] R8: 0xc04e4768:
[   88.640167] 4768  00000000 74737461 00000000 c0083204 00000000 c05335c0 00000000 c04e4784
[   88.640228] 4788  c04e4784 c0082e64 00000003 00000000 00000000 00000000 00000000 00000000
[   88.640289] 47a8  00000000 00000000 00000000 c00828f8 00000000 00000000 00000000 00000000
[   88.640319] 47c8  00000000 00000000 00000000 00000000 00000001 00000001 c04e47e0 c04e47e0
[   88.640380] 47e8  00000001 c68009a0 c04e47f0 c04e47f0 c0083cd0 00000000 c04e4800 c04e4800
[   88.640441] 4808  c0083abc 00000000 c04e4810 c04e4810 c050033c c04e1f64 c0083eb0 c04e483c
[   88.640472] 4828  00000000 0000003c 00000043 c68222e0 00000000 c0084170 00000000 00000000
[   88.640533] 4848  00000078 00008000 0000000a 00000000 00000000 c0535c34 c0086a94 c04dd474
[   88.640594] 
[   88.640594] R9: 0xc6800928:
[   88.640594] 0928  ffffffff ffffffff ffffffff ffffffff fffbffff ffffffff 65776f70 00302e72
[   88.640655] 0948  ffffffff ffffffff ffffffff fff7ffff ffffffff ffffffff 76697264 ff007265
[   88.640716] 0968  ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff c68009a0 c6800984
[   88.640747] 0988  c6800984 c04551e4 00000001 00000000 00000000 00000000 c68009a0 c68009a0
[   88.640808] 09a8  c68009a8 c68009a8 c04e47ec c6800980 c6825160 00000000 75626564 ffff0067
[   88.640869] 09c8  ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 75626564 ffff0067
[   88.640899] 09e8  ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 66757063 00716572
[   88.640960] 0a08  ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 66757063 00716572
[   88.641021] 
[   88.641021] R10: 0xc6847f04:
[   88.641021] 7f04  c0535c1c c68e6ea0 c04e4818 c04e47e8 c68009a8 c6847f84 c6847f4c c6847f50
[   88.641082] 7f24  c6847f38 c01b0640 c01b063c a0000013 ffffffff c01b0620 c6846000 c6847f74
[   88.641143] 7f44  c6847f50 c0083dc8 c01b062c c04e47ec c68009a0 c6846000 c68009a0 c0083cd0
[   88.641204] 7f64  00000000 c6847fc4 c6847f78 c00640f0 c0083cdc 00000000 c04e47ec 00000000
[   88.641235] 7f84  00000000 c6825160 c00684b8 c6847f90 c6847f90 c6847fc4 c6829ee0 00000002
[   88.641296] 7fa4  c0063fa4 c68009a0 00000000 00000000 00000000 c6847ff4 c6847fc8 c0068260
[   88.641357] 7fc4  c0063fb0 00000000 00000000 c6847fd0 c6847fd0 00000000 00000000 00000000
[   88.641387] 7fe4  00000000 00000000 c6847ff8 c00299d8 c00681e4 ffffffff ffffffff c6a52720
[   88.641448] Process suspend (pid: 8, stack limit = 0xc6846268)
[   88.641479] Stack: (0xc6847f38 to 0xc6848000)
[   88.641510] 7f20:                                                       c01b0620 c6846000
[   88.641540] 7f40: c6847f74 c6847f50 c0083dc8 c01b062c c04e47ec c68009a0 c6846000 c68009a0
[   88.641571] 7f60: c0083cd0 00000000 c6847fc4 c6847f78 c00640f0 c0083cdc 00000000 c04e47ec
[   88.641601] 7f80: 00000000 00000000 c6825160 c00684b8 c6847f90 c6847f90 c6847fc4 c6829ee0
[   88.641632] 7fa0: 00000002 c0063fa4 c68009a0 00000000 00000000 00000000 c6847ff4 c6847fc8
[   88.641662] 7fc0: c0068260 c0063fb0 00000000 00000000 c6847fd0 c6847fd0 00000000 00000000
[   88.641693] 7fe0: 00000000 00000000 00000000 c6847ff8 c00299d8 c00681e4 ffffffff ffffffff
[   88.641784] [<c01b063c>] (msmfb_suspend+0x1c/0x2c) from [<c0083dc8>] (early_suspend+0xf8/0x1a4)
[   88.641845] [<c0083dc8>] (early_suspend+0xf8/0x1a4) from [<c00640f0>] (worker_thread+0x14c/0x240)
[   88.641906] [<c00640f0>] (worker_thread+0x14c/0x240) from [<c0068260>] (kthread+0x88/0x90)
[   88.641967] [<c0068260>] (kthread+0x88/0x90) from [<c00299d8>] (kernel_thread_exit+0x0/0x8)
[   88.641998] Code: e5103038 e1a04000 e1a00003 e1a0e00f (e593f000) 
[   88.642059] ---[ end trace a026a5e39ad32e4a ]---

We have function names but not line numbers. Addr2line can solve that: in That line we have the program counter

[   88.635314] pc : [<c01b063c>]    lr : [<c01b0640>]    psr: a0000013

In order to translate it into source code line:

# arm-none-linux-gnueabi-addr2line -f -e vmlinux c01b063c
msmfb_suspend
/home/gnutoo/embedded/htcdream/SHR/kernel/linux/drivers/video/msm/msm_fb.c:485

二. gdb

来自Linus Torvalds的讨论:
https://groups.google.com/group/linux.kernel/browse_thread/thread/b70bffe9015a8c41/ed9c0a0cfcd31111[/url]

又，http://kerneltrap.org/Linux/Further_Oops_Insights

例如这样的一个Oops：
            Oops: 0000 [#1] PREEMPT SMP
            Modules linked in: capidrv kernelcapi isdn slhc ipv6 loop dm_multipath snd_ens1371 gameport snd_rawmidi snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd parport_pc floppy parport pcnet32 soundcore mii pcspkr snd_page_alloc ac i2c_piix4 i2c_core button power_supply sr_mod sg cdrom ata_piix libata dm_snapshot dm_zero dm_mirror dm_mod BusLogic sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd

            Pid: 1726, comm: kstopmachine Not tainted (2.6.24-rc3-module #2)
            EIP: 0060:[<c04e53d6>] EFLAGS: 00010092 CPU: 0
            EIP is at list_del+0xa/0x61
            EAX: e0c3cc04 EBX: 00000020 ECX: 0000000e EDX: dec62000
            ESI: df6e8f08 EDI: 000006bf EBP: dec62fb4 ESP: dec62fa4
               DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
            Process kstopmachine (pid: 1726, ti=dec62000 task=df8d2d40 task.ti=dec62000)
            Stack: 000006bf dec62fb4 c04276c7 00000020 dec62fbc c044ab4c dec62fd0 c045336c
                     df6e8f08 c04532b4 00000000 dec62fe0 c043deb0 c043de75 00000000 00000000
                     c0405cdf df6e8eb4 00000000 00000000 00000000 00000000 00000000
            Call Trace:
               [<c0406081>] show_trace_log_lvl+0x1a/0x2f
               [<c0406131>] show_stack_log_lvl+0x9b/0xa3
               [<c04061dc>] show_registers+0xa3/0x1df
               [<c0406437>] die+0x11f/0x200
               [<c0613cba>] do_page_fault+0x533/0x61a
               [<c06123ea>] error_code+0x72/0x78
               [<c044ab4c>] __unlink_module+0xb/0xf
               [<c045336c>] do_stop+0xb8/0x108
               [<c043deb0>] kthread+0x3b/0x63
               [<c0405cdf>] kernel_thread_helper+0x7/0x10
               =======================
            Code: 6b c0 e8 2e 7e f6 ff e8 d1 16 f2 ff b8 01 00 00 00 e8 aa 1c f4 ff 89 d8 83 c4 10 5b 5d c3 90 90 90 55 89 e5 53 83 ec 0c 8b 48 04 <8b> 11 39 c2 74 18 89 54 24 08 89 44 24 04 c7 04 24 be 32 6b c0
            EIP: [<c04e53d6>] list_del+0xa/0x61 SS:ESP 0068:dec62fa4
            note: kstopmachine[1726] exited with preempt_count 1

      1, 有自己编译的vmlinux：使用gdb

         编译时打开complie with debug info选项。

         注意这行：

            EIP is at list_del+0xa/0x61

         这告诉我们，list_del函数有0x61这么大，而Oops发生在0xa处。那么我们先看一下list_del从哪里开始：

            # grep list_del /boot/System.map-2.6.24-rc3-module
            c10e5234 T plist_del
            c10e53cc T list_del
            c120feb6 T klist_del
            c12d6d34 r __ksymtab_list_del
            c12dadfc r __ksymtab_klist_del
            c12e1abd r __kstrtab_list_del
            c12e9d03 r __kstrtab_klist_del

         于是我们知道，发生Oops时的EIP值是：

            c10e53cc + 0xa  == c10e53d6

         然后用gdb查看：

            # gdb /home/arc/build/linux-2.6/vmlinux
            (gdb) b *0xc10e53d6
            Breakpoint 1 at 0xc10e53d6: file /usr/src/linux-2.6.24-rc3/lib/list_debug.c, line 64.

         看，gdb直接就告诉你在哪个文件、哪一行了。

         gdb中还可以这样：

            # gdb Sources/linux-2.6.24/vmlinux
            (gdb) l *do_fork+0x1f
            0xc102b7ac is in do_fork (kernel/fork.c:1385).
            1380
            1381 static int fork_traceflag(unsigned clone_flags)
            1382 {
            1383          if (clone_flags & CLONE_UNTRACED)
            1384                   return 0;
            1385          else if (clone_flags & CLONE_VFORK) {
            1386                   if (current->ptrace & PT_TRACE_VFORK)
            1387                         return PTRACE_EVENT_VFORK;
            1388          } else if ((clone_flags & CSIGNAL) != SIGCHLD) {
            1389                   if (current->ptrace & PT_TRACE_CLONE)
            (gdb)

         也可以直接知道line number。

         或者：

            (gdb) l *(0xffffffff8023eaf0 + 0xff)  /* 出错函数的地址加上偏移 */

      2, 没有自己编译的vmlinux： TIPS

         如果在lkml或bugzilla上看到一个Oops，而自己不能重现，那就只能反汇编以"Code:"开始的行。这样可以尝试定位到
         源代码中。

         注意，Oops中的Code:行，会把导致Oops的第一条指令，也就是EIP的值的第一个字节，用尖括号<>括起来。但是，有些
         体系结构(例如常见的x86)指令是不等长的(不一样的指令可能有不一样的长度)，所以要不断的尝试(trial-and-error)。

         Linus通常使用一个小程序，类似这样：

            const char array[] = "\xnn\xnn\xnn...";
            int main(int argc, char *argv[])
            {
                     printf("%p\n", array);
                     *(int *)0 = 0;
            }

e.g. /*{{{*/ /* 注意， array一共有从array[0]到array[64]这65个元素，其中出错的那个操作码<8b> == arry[43] */
#include <stdio.h>
#include <stdlib.h>

const char array[] ="\x6b\xc0\xe8\x2e\x7e\xf6\xff\xe8\xd1\x16\xf2\xff\xb8\x01\x00\x00\x00\xe8\xaa\x1c\xf4\xff\x89\xd8\x83\xc4\x10\x5b\x5d\xc3\x90\x90\x90\x55\x89\xe5\x53\x83\xec\x0c\x8b\x48\x04\x8b\x11\x39\xc2\x74\x18\x89\x54\x24\x08\x89\x44\x24\x04\xc7\x04\x24\xbe\x32\x6b\xc0";
int main(int argc, char *argv[])
{
      printf("%p\n", array);
      *(int *)0 = 0;
}
/*}}}*/

         用gcc -g编译，在gdb里运行它：

            [arc@dhcp-cbjs05-218-251 ~]$ gdb hello
            GNU gdb Fedora (6.8-1.fc9)
            Copyright (C) 2008 Free Software Foundation, Inc.
            License GPLv3+: GNU GPL version 3 or later <[url]http://gnu.org/licenses/gpl.html[/url]>
            This is free software: you are free to change and redistribute it.
            There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
            and "show warranty" for details.
            This GDB was configured as "x86_64-redhat-linux-gnu"...
            (no debugging symbols found)
            (gdb) r
            Starting program: /home/arc/hello
            0x80484e0

            Program received signal SIGSEGV, Segmentation fault.

         注意，这时候就可以反汇编0x80484e0这个地址：

            (gdb) disassemble 0x80484e0
            Dump of assembler code for function array:
            0x080484e0 <array+0>: imul $0xffffffe8,%eax,%eax
            0x080484e3 <array+3>: jle,pn 0x80484dc <__dso_handle+20>
            0x080484e6 <array+6>: ljmp *<internal disassembler error>
            0x080484e8 <array+8>: rcll (%esi)
            0x080484ea <array+10>:  repnz (bad)
            0x080484ec <array+12>:  mov $0x1,%eax
            0x080484f1 <array+17>:  call 0x7f8a1a0
            0x080484f6 <array+22>:  mov %ebx,%eax
            0x080484f8 <array+24>:  add $0x10,%esp
            0x080484fb <array+27>:  pop %ebx
            0x080484fc <array+28>:  pop %ebp
            0x080484fd <array+29>:  ret
            0x080484fe <array+30>:  nop
            0x080484ff <array+31>:  nop
            0x08048500 <array+32>:  nop
            0x08048501 <array+33>:  push %ebp
            0x08048502 <array+34>:  mov %esp,%ebp
            0x08048504 <array+36>:  push %ebx
            0x08048505 <array+37>:  sub $0xc,%esp
            0x08048508 <array+40>:  mov 0x4(%eax),%ecx
            0x0804850b <array+43>:  mov (%ecx),%edx
            0x0804850d <array+45>:  cmp %eax,%edx
            0x0804850f <array+47>:  je    0x8048529
            0x08048511 <array+49>:  mov %edx,0x8(%esp)
            0x08048515 <array+53>:  mov %eax,0x4(%esp)
            0x08048519 <array+57>:  movl $0xc06b32be,(%esp)
            0x08048520 <array+64>:  add %ah,0xa70
            End of assembler dump.
            (gdb)

      OK, 现在你知道出错的那条指令是array[43]，也就是mov (%ecx),%edx，也就是说，(%ecx)指向了一个错误内存地址。

补充：

为了使汇编代码和C代码更好的对应起来， Linux内核的Kbuild子系统提供了这样一个功能：任何一个C文件都可以单独编译成汇编文件，例如：

make path/to/the/sourcefile.s

例如我想把kernel/sched.c编译成汇编，那么：

make kernel/sched.s V=1

或者：

make kernel/sched.lst V=1

   编译出*.s文件

   有时侯需要对*.s文件进行分析，以确定BUG所在的位置。对任何一个内核build目录下的*.c文件，都可以
   直接编译出*.s文件。

   # make drivers/net/e100.s V=1

   而对于自己写的module，就需要在Makefile中有一个灵活的target写法：

# cat Makefile
obj-m := usb-skel.o
KDIR := /lib/modules/`uname -r`/build
traget := modules

default:
make -C $(KDIR) M=$(shell pwd) $(target)
clean:
rm -f *.o *.ko .*.cmd *.symvers *.mod.c
rm -rf .tmp_versions

# make target=usb-skel.s V=1

   这样，kbuild系统才知道你要make的目标不是modules，而是usb-skel.s。

另外，内核源代码目录的./scripts/decodecode文件是用来解码Oops的：

./scripts/decodecode < Oops.txt

声明：以下部分来源于http://blog.chinaunix.net/uid-22335998-id-1774894.html

三从自己编译的内核模块出错信息中获取代码行

以ldd3中提供的misc-modules/faulty.c为例。主要以faulty_write函数作分析。
（1）由于作者提供的函数代码就一样，过于简单，我这里简单加上一些代码（也就是判断和赋值），如下：

ssize_t faulty_write (struct file *filp, const char __user *buf, size_t count, loff_t *pos) { if(count > 0x100) count = 0x100; *(int *)0 = 0; return count; }

（2）编译该模块，并且mknod /dev/faulty
（3）向该模块写入数据：echo 1 > /dev/faulty, 内核OOPS，信息如下：

<1>BUG: unable to handle kernel NULL pointer dereference at virtual address 00000000 printing eip: f8e8000e *pde = 00000000 Oops: 0002 [#3] SMP Modules linked in: faulty autofs4 hidp rfcomm l2cap ...... //此处省略若干字符 CPU: 1 EIP: 0060:[<f8e8000e>] Not tainted VLI EFLAGS: 00010283 (2.6.18.3 #2) EIP is at faulty_write+0xe/0x19 [faulty] eax: 00000001 ebx: f4f6ca40 ecx: 00000001 edx: b7c2d000 esi: f8e80000 edi: b7c2d000 ebp: 00000001 esp: f4dc5f84 ds: 007b es: 007b ss: 0068 Process bash (pid: 6084, ti=f4dc5000 task=f7c8d4d0 task.ti=f4dc5000) Stack: c1065914 f4dc5fa4 f4f6ca40 fffffff7 b7c2d000 f4dc5000 c1065f06 f4dc5fa4 00000000 00000000 00000000 00000001 00000001 c1003d0b 00000001 b7c2d000 00000001 00000001 b7c2d000 bfd40aa8 ffffffda 0000007b c100007b 00000004 Call Trace: [<c1065914>] vfs_write+0xa1/0x143 [<c1065f06>] sys_write+0x3c/0x63 [<c1003d0b>] syscall_call+0x7/0xb Code: Bad EIP value. EIP: [<f8e8000e>] faulty_write+0xe/0x19 [faulty] SS:ESP 0068:f4dc5f84

其中，我们应该关注的信息是第一行红色标出部分：告诉我们操作了NULL指针。其次，就是第二行红色部分：EIP is at faulty_write+0xe/0x19。这个出错信息告诉我们EIP指针出现问题的地方时faulty_write函数，而且指出了是faulty模块。
同时，faulty_write+0xe/0x19的后半部分0xe/0x19，说明该函数的大小时0x019,出错位置是在0x0e。这两个值应该值得都是汇编代码的值。
（4）将faulty模块反汇编出汇编代码：

         objdump -d faulty.ko > faulty.s
      或
         objdump -d faulty.o > faulty.s
然后，我们打开faulty.s文件。由于我们需要关注的部分正好在文件的前面，因此我这里只贴出文件的前面一部分内容：

faulty.o: file format elf32-i386 Disassembly of section .text: 00000000 <faulty_write>: 0: 81 f9 00 01 00 00 cmp $0x100,%ecx 6: b8 00 01 00 00 mov $0x100,%eax b: 0f 46 c1 cmovbe %ecx,%eax e: c7 05 00 00 00 00 00 movl $0x0,0x0 15: 00 00 00 18: c3 ret 00000019 <cleanup_module>: 19: a1 00 00 00 00 mov 0x0,%eax 1e: ba 00 00 00 00 mov $0x0,%edx 23: e9 fc ff ff ff jmp 24 <cleanup_module+0xb> 00000028 <faulty_init>: 28: a1 00 00 00 00 mov 0x0,%eax 2d: b9 00 00 00 00 mov $0x0,%ecx 32: ba 00 00 00 00 mov $0x0,%edx 37: e8 fc ff ff ff call 38 <faulty_init+0x10> 3c: 85 c0 test %eax,%eax 3e: 78 13 js 53 <faulty_init+0x2b> 40: 83 3d 00 00 00 00 00 cmpl $0x0,0x0 47: 74 03 je 4c <faulty_init+0x24> 49: 31 c0 xor %eax,%eax 4b: c3 ret 4c: a3 00 00 00 00 mov %eax,0x0 51: 31 c0 xor %eax,%eax 53: c3 ret

由以上汇编代码可以看出，faulty_write函数的大小确实是0x18 -0x00 +1 = 0x19. 那么EIP指针出问题的地方是0x0e处，代码为：

e: c7 05 00 00 00 00 00 movl $0x0,0x0

这行汇编代码就是将0值保存到0地址的位置。那么很显然是非法的。这一行对应的C 代码应该就是：
*(int *)0 = 0;

（5）以上是对模块出错信息的分析。不过也有一定的局限。
首先就是EIP出错的位置正好在本模块内部，这样可以在本模块定位问题；
其次，要求一定的汇编基础，特别是当一个函数的代码比较多时，对应的汇编代码也比较大，如何准确定位到C代码行需要一定的经验和时间。

jiayinjia1983116

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
linux kernel oops

如何快速的定位oops一. addr2line 工具 Addr2line 工具（它是标准的 GNU Binutils 中的一部分）是一个可以将指令的地址和可执行映像转换成文件名、函数名和源代码行数的工具。这种功能对于将跟踪地址转换成更有意义的内容来说简直是太棒了。有了 addr2line 工具，遇到oops时，我们就可以找到当前PC（EIP）所在地址，然后就可转换为相应的C文件里具
复制链接

扫一扫