kernel crash analysis

1. write one self trigger ramdump method

#echo c > /proc/sysrq-trigger

2. load kernel dump file to crash analyzer

crash vmlinux /home/xxxxx/debug/K1939EL.RAM

or

crash -m phys_base=offset(0x80000000) vmlinux K0342EL.RAM

 

3.setup crash environment

  mkdir debug

 cd debug

 wget http://people.redhat.com/anderson/crash-7.0.0.tar.gz

 tar -zxvf crash-7.0.0.tar.gz
 cd crash-7.0.0

 make target=ARM

 sudo make install crash

 sudo make extensions ( and it will generate some extension lib)

 

4. some commands

   a. bt pid | task--- display stack backtrace

bt
PID: 0      TASK: c06913a8  CPU: 0   COMMAND: "swapper/0"
 #0 [<c0489da4>] (__schedule) from [<c048a47c>]
 #1 [<c048a3fc>] (schedule) from [<c048a6d4>]
 #2 [<c048a6b0>] (schedule_preempt_disabled) from [<c0010a48>]
 #3 [<c0010974>] (cpu_idle) from [<c04795d4>]
 #4 [<c0479564>] (rest_init) from [<c06397c4>]
 #5 [<c063951c>] (start_kernel) from [<8000803c>]

   b. log or log > 1.log to get log

  c. objdump to generate asm code from elf

   ./../../../../prebuilt/linux-x86/toolchain/arm-eabi-4.4.3/bin/arm-eabi-objdump -D vmlinux > kernel.asm

 d.read data from the memory
 #rd

 e.# irq

 f. ps to list all threads,

 g. task to list all task information

h. help to get all manual information

 

5. example.

find corrupt, and we will analyze pc(r15), sp(r13), fp(r11).

 

[  250.553810:0] Internal error: Oops: 805 [#1] PREEMPT SMP ARM
[  250.559479:0] Modules linked in: bcmdhd mali ump
[  250.564216:0] CPU: 0    Tainted: G        W     (3.4.5-g3d90f8c-dirty #27)
[  250.571118:0] PC is at sysrq_handle_crash+0x20/0x2c
[  250.576086:0] LR is at __handle_sysrq+0xa8/0x154
[  250.580716:0] pc : [<c01f0600>]    lr : [<c01f0c38>]    psr: 60000093
[  250.580725:0] sp : e5347ec0  ip : e5347ed0  fp : e5347ecc
[  250.592650:0] r10: e5347f78  r9 : ee98d10c  r8 : 00000000
[  250.598139:0] r7 : 60000013  r6 : 00000063  r5 : 00000007  r4 : c06ab6f0
[  250.604851:0] r3 : 00000001  r2 : 00000000  r1 : 60000093  r0 : 00000063
[  250.611645:0] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  250.619135:0] Control: 10c53c7d  Table: 9e09406a  DAC: 00000015
[  250.625064:0]
[  250.625069:0] PC: 0xc01f0580:
[  250.629771:0] 0580  1a000001 eb0a6ca0 ea000006 eb0a6c9e e1a00007 e1a01006 ebffe0a0 ea000001
[  250.638194:0] 05a0  e1a00006 ebffdb8f e1a00004 e89da9f8 c07143ec c06ab444 e1a0c00d e92dd800
[  250.646699:0] 05c0  e24cb004 e59f3010 e5932000 e3520000 05d30004 13a00001 e89da800 c0689ee0
[  250.655207:0] 05e0  e1a0c00d e92dd800 e24cb004 e59f2014 e3a03001 e5823000 f57ff04f e3a02000
[  250.663643:0] 0600  e5c23000 e89da800 c06e5dac e1a0c00d e92dd830 e24cb004 e59f401c e2405030
[  250.672072:0] 0620  e59f0018 e3a03007 e1a01005 e5843000 eb0a4e54 e5845000 e89da830 c06a00b0
[  250.680576:0] 0640  c05b8641 e1a0c00d e92dd830 e24cb004 e5904000 e1a05000 eb01caf8 e2840024
[  250.689083:0] 0660  ebf96722 e1a00005 eb01cab3 e1a00004 ebfb3c89 e89da83