linux内核调试

1.使用kdump+debuginfo+crash
[root@localhost 127.0.0.1-2014-06-19-03:01:35]# crash /usr/lib/debug/lib/modules/2.6.32-358.el6.x86_64/vmlinux vmcore


crash 6.1.0-1.el6
Copyright (C) 2002-2012  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...


      KERNEL: /usr/lib/debug/lib/modules/2.6.32-358.el6.x86_64/vmlinux
    DUMPFILE: vmcore  [PARTIAL DUMP]
        CPUS: 40
        DATE: Thu Jun 19 03:01:29 2014
      UPTIME: 00:25:22
LOAD AVERAGE: 0.29, 0.10, 0.04
       TASKS: 836
    NODENAME: localhost.localdomain
     RELEASE: 2.6.32-358.el6.x86_64
     VERSION: #1 SMP Wed Jun 18 23:51:42 CST 2014
     MACHINE: x86_64  (2800 Mhz)
      MEMORY: 64 GB
       PANIC: "Oops: 0002 [#1] SMP " (check log for details)
         PID: 8324
     COMMAND: "bash"
        TASK: ffff880816acaae0  [THREAD_INFO: ffff8808100fc000]
         CPU: 2
       STATE: TASK_RUNNING (PANIC)


crash> bt
PID: 8324   TASK: ffff880816acaae0  CPU: 2   COMMAND: "bash"
 #0 [ffff8808100fd9e0] machine_kexec at ffffffff81035b7b
 #1 [ffff8808100fda40] crash_kexec at ffffffff810c0db2
 #2 [ffff8808100fdb10] oops_end at ffffffff815111d0
 #3 [ffff8808100fdb40] no_context at ffffffff81046bfb
 #4 [ffff8808100fdb90] __bad_area_nosemaphore at ffffffff81046e85
 #5 [ffff8808100fdbe0] bad_area at ffffffff81046fae
 #6 [ffff8808100fdc10] __do_page_fault at ffffffff81047760
 #7 [ffff8808100fdd30] do_page_fault at ffffffff8151311e
 #8 [ffff8808100fdd60] page_fault at ffffffff815104d5
    [exception RIP: sysrq_handle_crash+22]
    RIP: ffffffff8133d626  RSP: ffff8808100fde18  RFLAGS: 00010096
    RAX: 0000000000000010  RBX: 0000000000000063  RCX: 0000000000004e93
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 0000000000000063
    RBP: ffff8808100fde18   R8: 0000000000000000   R9: 00007f62938b2700
    R10: 0000000000000000  R11: 0000000000000000  R12: 0000000000000000
    R13: ffffffff81affea0  R14: 0000000000000286  R15: 0000000000000004
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #9 [ffff8808100fde20] __handle_sysrq at ffffffff8133d8e2
#10 [ffff8808100fde70] write_sysrq_trigger at ffffffff8133d99e
#11 [ffff8808100fdea0] proc_reg_write at ffffffff811e95ae
#12 [ffff8808100fdef0] vfs_write at ffffffff81180f98
#13 [ffff8808100fdf30] sys_write at ffffffff81181891
#14 [ffff8808100fdf80] system_call_fastpath at ffffffff8100b072
    RIP: 00000036288db650  RSP: 00007fff5711b118  RFLAGS: 00010206
    RAX: 0000000000000001  RBX: ffffffff8100b072  RCX: 0000000000000000
    RDX: 0000000000000002  RSI: 00007f62938c2000  RDI: 0000000000000001
    RBP: 00007f62938c2000   R8: 000000000000000a   R9: 00007f62938b2700
    R10: 0000000000000000  R11: 0000000000000246  R12: 0000000000000002
    R13: 0000003628b8d780  R14: 0000000000000002  R15: 0000003628b8d780
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b
crash> l *(sysrq_handle_crash+22)
0xffffffff8133d626 is in sysrq_handle_crash (drivers/char/sysrq.c:130).
125     {
126             char *killer = NULL;
127     
128             panic_on_oops = 1;      /* force panic */
129             wmb();
130             *killer = 1;
131     }
132     static struct sysrq_key_op sysrq_crash_op = {
133             .handler        = sysrq_handle_crash,
134             .help_msg       = "Crash",
crash> 








2.反汇编命令


# objdump -dS submiter_test.ko


submiter_test.ko:     file format elf64-x86-64




Disassembly of section .text:


0000000000000000 <value_to_buf>:


        bio_put(bio);
}


int value_to_buf(char *buf, int *pos, unsigned long value)
{
   0:   55                      push   %rbp 
   1:   48 89 e5                mov    %rsp,%rbp
   4:   48 81 ec 90 00 00 00    sub    $0x90,%rsp
   b:   e8 00 00 00 00          callq  10 <value_to_buf+0x10>
  10:   65 48 8b 04 25 28 00    mov    %gs:0x28,%rax
  17:   00 00 
  19:   48 89 45 f8             mov    %rax,-0x8(%rbp)
  1d:   31 c0                   xor    %eax,%eax
        char tmp[128];
        char v;
        int j = 0; 
        unsigned long t = value;


        if (!pos || !buf)
  1f:   48 85 f6                test   %rsi,%rsi
  22:   0f 84 b8 00 00 00       je     e0 <value_to_buf+0xe0>
  28:   48 85 ff                test   %rdi,%rdi
  2b:   0f 84 af 00 00 00       je     e0 <value_to_buf+0xe0>
...


可根据下面的debug函数栈中指示的位置去定位bug。


debug 函数栈


对下面的解释：
BUG：访问空指针导致内核崩溃，访问位置：<1>IP: [<ffffffffa001618a>] bvecs_to_buf+0x7a/0xd0 [submiter_test]
意为[submiter_test]模块中的bvecs_to_buf函数长度为0xd0，访问空指针发生在0x7a.


<1>BUG: unable to handle kernel NULL pointer dereference at (null)
<1>IP: [<ffffffffa001618a>] bvecs_to_buf+0x7a/0xd0 [submiter_test]
<4>PGD 0 
<4>Oops: 0000 [#1] SMP
<4>last sysfs file: /sys/kernel/mm/ksm/run
<4>CPU 0 
<4>Modules linked in: submiter_test(U) ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge autofs4 sunrpc 8021q garp stp llc ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 fuse vmhgfs(U) vsock(U) ext3 jbd vhost_net macvtap macvlan tun uinput ppdev parport_pc parport snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc e1000 microcode vmware_balloon sg vmci(U) i2c_piix4 i2c_core shpchp ext4 mbcache jbd2 sr_mod cdrom ahci sd_mod crc_t10dif mptspi mptscsih mptbase scsi_transport_spi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] <4><4>Pid: 3172, comm: sshd Not tainted 2.6.32-358.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
<4>RIP: 0010:[<ffffffffa001618a>]  [<ffffffffa001618a>] bvecs_to_buf+0x7a/0xd0 [submiter_test]
<4>RSP: 0018:ffff88000b403c88  EFLAGS: 00010202
<4>RAX: 0000000000000000 RBX: ffff88003d851000 RCX: 0000000000000000
<4>RDX: 0000000000000000 RSI: ffff88000b403ca4 RDI: ffff88003d851000
<4>RBP: ffff88000b403cd8 R08: ffffea0000d98720 R09: 0000000000000000
<4>R10: ffff88000241e1c0 R11: 0000000000000000 R12: ffff88003e9d61d0
<4>R13: 0000000000000000 R14: 0000000000000001 R15: ffff88000b403ca4
<4>FS:  00007f5f94e367c0(0000) GS:ffff88000b400000(0000) knlGS:0000000000000000
<4>CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>CR2: 0000000000000000 CR3: 000000003e25d000 CR4: 00000000000407f0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process sshd (pid: 3172, threadinfo ffff88003de8c000, task ffff88003deeb540)
<4>Stack:
<4> 0000000000000080 ffff88003dfd9a80 ffff88000b403ca8 00000000811bb1a5
<4><d> ffff88000b403cb8 ffff88003ddad6c0 ffff88003d851000 0000000000000000
<4><d> ffff88003dfd9a80 0000000000000000 ffff88000b403d08 ffffffffa00169ab
<4>Call Trace:
<4> <IRQ>
<4> [<ffffffffa00169ab>] test_write_bio_endio+0x6b/0xe0 [submiter_test]
<4> [<ffffffff811b920d>] bio_endio+0x1d/0x40
<4> [<ffffffff8125ab9b>] req_bio_endio+0x9b/0xe0
<4> [<ffffffff8125c647>] blk_update_request+0x107/0x490
<4> [<ffffffff8125c9f7>] blk_update_bidi_request+0x27/0xa0
<4> [<ffffffff8125de3f>] blk_end_bidi_request+0x2f/0x80
<4> [<ffffffff8125dee0>] blk_end_request+0x10/0x20
<4> [<ffffffff8137a61f>] scsi_io_completion+0xaf/0x6c0
<4> [<ffffffff81371652>] scsi_finish_command+0xc2/0x130
<4> [<ffffffff8137ad95>] scsi_softirq_done+0x145/0x170
<4> [<ffffffff81263545>] blk_done_softirq+0x85/0xa0
<4> [<ffffffff81076fb1>] __do_softirq+0xc1/0x1e0
<4> [<ffffffff8100c1cc>] call_softirq+0x1c/0x30
<4> <EOI>
<4> [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
<4> [<ffffffff81076e3a>] local_bh_enable+0x9a/0xb0
<4> [<ffffffff8144c76c>] dev_queue_xmit+0x16c/0x550
<4> [<ffffffff81484f40>] ? ip_finish_output+0x0/0x310
<4> [<ffffffff8148507c>] ip_finish_output+0x13c/0x310
<4> [<ffffffff81485308>] ip_output+0xb8/0xc0
<4> [<ffffffff814845cf>] ? __ip_local_out+0x9f/0xb0
<4> [<ffffffff81484605>] ip_local_out+0x25/0x30
<4> [<ffffffff81484ae0>] ip_queue_xmit+0x190/0x420
<4> [<ffffffff81197160>] ? __pollwait+0x0/0xf0
<4> [<ffffffff814997ce>] tcp_transmit_skb+0x3fe/0x7b0
<4> [<ffffffff8149bb8b>] tcp_write_xmit+0x1fb/0xa20
<4> [<ffffffff8149c540>] __tcp_push_pending_frames+0x30/0xe0
<4> [<ffffffff8148bdbe>] tcp_push+0x6e/0x90
<4> [<ffffffff8148cdbc>] tcp_sendmsg+0x64c/0xa20
<4> [<ffffffff81437b9b>] sock_aio_write+0x19b/0x1c0
<4> [<ffffffff8132919c>] ? n_tty_read+0x34c/0x990
<4> [<ffffffff81180c9a>] do_sync_write+0xfa/0x140
<4> [<ffffffff81096c80>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffff81228bdf>] ? selinux_file_permission+0xbf/0x150
<4> [<ffffffff8121baf6>] ? security_file_permission+0x16/0x20
<4> [<ffffffff81181064>] vfs_write+0x184/0x1a0
<4> [<ffffffff81181891>] sys_write+0x51/0x90
<4> [<ffffffff810dc565>] ? __audit_syscall_exit+0x265/0x290
<4> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
<4>Code: e0 4c 8b 6d e8 4c 8b 75 f0 4c 8b 7d f8 c9 c3 4c 8d 7d cc 31 c0 45 31 f6 48 c1 e0 04 4c 89 fe 48 89 df 49 8d 4c 05 00 41 83 c6 01 <48> 8b 11 48 89 4d b8 e8 6a fe ff ff 8b 75 cc 48 c7 c7 2f 6b 01
<1>RIP  [<ffffffffa001618a>] bvecs_to_buf+0x7a/0xd0 [submiter_test]
<4> RSP <ffff88000b403c88>
<4>CR2: 0000000000000000