使用mdb调试Solaris内核crash文件

Working with Kernel Cores The most common type of kernel debug target is a core file, saved from a prior system crash. In the following sections, we highlight some of the introductory steps as used with mdb to explore a kernel core image. 14.1.1. Locating and Attaching the Target If a system has crashed, then we should have a core image saved in /var/crash on the target machine. The mdb debugger should be invoked from a system with the same architecture and Solaris revision as the crash image. The first steps are to locate the appropriate saved image and then to invoke mdb. # cd /var/crash/nodename # ls bounds unix.1 unix.3 unix.5 unix.7 vmcore.1 vmcore.3 vmcore.5 vmcore.7 unix.0 unix.2 unix.4 unix.6 vmcore.0 vmcore.2 vmcore.4 vmcore.6 # mdb -k unix.7 vmcore.7 Loading modules: [ unix krtld$c genunix specfs dtrace ufs ip sctp usba uhci s1394 fcp fctl nca lofs zfs random nfs audiosup sppp crypto md fcip logindmux ptm ipc ] > 14.1.2. Examining Kernel Core Summary Information The kernel core contains important summary information from which we can extract the following: Revision of the kernel Hostname CPU and platform architecture of the system Panic string Module causing the panic We can use the ::showrev and ::status dcmds to extract this information. > ::showrev Hostname: zones-internal Release: 5.11 Kernel architecture: i86pc Application architecture: i386 Kernel version: SunOS 5.11 i86pc snv_27 Platform: i86pc > ::status debugging crash dump vmcore.2 (32-bit) from zones-internal operating system: 5.11 snv_27 (i86pc) panic message: BAD TRAP: type=e (#pf Page fault) rp=d2a587c8 addr=0 occurred in module "unix" due to a NULL pointer dereference dump content: kernel pages only > ::panicinfo cpu 0 thread d2a58de0 message BAD TRAP: type=e (#pf Page fault) rp=d2a587c8 addr=0 occurred in module "unix" due to a NULL pointer dereference gs fe8301b0 fs fec30000 es fe8d0160 ds d9820160 edi 0 esi dc062298 ebp d2a58828 esp d2a58800 ebx de453000 edx d2a58de0 ecx 1 eax 0 trapno e err 2 eip fe82ca58 cs 158 eflags 10282 uesp fe89ab0d ss 0 gdt fec1f2f002cf idt fec1f5c007ff ldt 140 task 150 cr0 8005003b cr2 0 cr3 4cb3000 cr4 6d8 14.1.3. Examining the Message Buffer The kernel keeps a cyclic buffer of the recent kernel messages. In this buffer we can observe the messages up to the time of the panic. The ::msgbuf dcmd shows the contents of the buffer. > ::msgbuf MESSAGE /pseudo/zconsnex@1/zcons@5 (zcons5) online /pseudo/zconsnex@1/zcons@6 (zcons6) online /pseudo/zconsnex@1/zcons@7 (zcons7) online pseudo-device: ramdisk1024 ... panic[cpu0]/thread=d2a58de0: BAD TRAP: type=e (#pf Page fault) rp=d2a587c8 addr=0 occurred in module "unix" due to a NULL pointer dereference sched: #pf Page fault Bad kernel fault at addr=0x0 pid=0, pc=0xfe82ca58, sp=0xfe89ab0d, eflags=0x10282 cr0: 8005003b<pg,wp,ne,et,ts,mp,pe> cr4: 6d8<xmme,fxsr,pge,mce,pse,de> cr2: 0 cr3: 4cb3000 gs: fe8301b0 fs: fec30000 es: fe8d0160 ds: d9820160 edi: 0 esi: dc062298 ebp: d2a58828 esp: d2a58800 ebx: de453000 edx: d2a58de0 ecx: 1 eax: 0 trp: e err: 2 eip: fe82ca58 cs: 158 efl: 10282 usp: fe89ab0d ss: 0 ... 14.1.4. Obtaining a Stack Trace of the Running Thread We can obtain a stack backtrace of the current thread by using the $C command. Note that the displayed arguments to each function are not necessarily accurate. On each platform, the meaning of the shown arguments is as follows: SPARC. The values of the arguments if they are available from a saved stack frame, assuming they are not overwritten by use of registers during the called function. With SPARC architectures, a function's input argument registers are sometimes saved on the way out of a functionif the input registers are reused during the function, then values of the input arguments are overwritten and lost. x86. Accurate values of the input arguments. Input arguments are always saved onto the stack and can be accurately displayed x64. The values of the arguments, assuming they are available. As with the SPARC architectures, input arguments are passed in registers and may be overwritten. > $C d2a58828 atomic_add_32+8(0) d2a58854 nfs4_async_inactive+0x3b(dc1c29c0, 0) d2a58880 nfs4_inactive+0x41() d2a5889c fop_inactive+0x15(dc1c29c0, 0) d2a588b0 vn_rele+0x4b(dc1c29c0) d2a588c0 snf_smap_desbfree+0x59(dda94080) d2a588dc dblk_lastfree_desb+0x13(de45b520, d826fb40) d2a588f4 dblk_decref+0x4e(de45b520, d826fb40) d2a58918 freemsg+0x69(de45b520) d2a5893c FreeTxSwPacket+0x3b(d38b84f0) d2a58968 CleanTxInterrupts+0xb4(d2f9cac0) d2a589a4 e1000g_send+0xf6(d2f9cac0, d9ffba00) d2a589c0 e1000g_m_tx+0x22() d2a589dc dls_tx+0x16(d4520f68, d9ffba00) d2a589f4 str_mdata_fastpath_put+0x1e(d3843f20, d9ffba00) d2a58a40 tcp_send_data+0x62d(db0ecac0, d97ee250, d9ffba00) d2a58aac tcp_send+0x6b6(d97ee250, db0ecac0, 564, 28, 14, 0) d2a58b40 tcp_wput_data+0x622(db0ecac0, 0, 0) d2a58c28 tcp_rput_data+0x2560(db0ec980, db15bd20, d2d45f40) d2a58c40 tcp_input+0x3c(db0ec980, db15bd20, d2d45f40) d2a58c78 squeue_enter_chain+0xe9(d2d45f40, db15bd20, db15bd20, 1, 1) d2a58cec ip_input+0x658(d990e554, d3164010, 0, e) d2a58d40 i_dls_link_ether_rx+0x156(d4523db8, d3164010, db15bd20) d2a58d70 mac_rx+0x56(d3520200, d3164010, db15bd20) d2a58dac e1000g_intr+0xa6(d2f9cac0, 0) d2a58ddc intr_thread+0x122() 14.1.5. Which Process?