Rate this page
del.icio.us
Digg
Take note that:
- You should use
-ivhrather than-Uvhwhen installing the kernel package. This will preserve the older version of the kernel installed so that you can revert back to a known working version of the kernel should you encounter any problems with the new version. - The kernel-debuginfo package for an older kernel can safely remain installed when installing a newer version. The kernel-debuginfo must match the kernel version, variant, and architecture that created the
vmcore. See thefile ./vmcoreandstrings vmcore | fgrep -m1 ‘Linux ‘commands in the above output. - In Red Hat Enterprise Linux 5, the
vmlinuxkernel-debuginfo package is divided into two packages:kernel-debuginfo-version.arch.rpmandkernel-debuginfo-common-version.arch.rpm. Both are required in order to perform crash dump analysis on the Red Hat Enterprise Linux 5 kernels.
Run crash
When crash is run on a vmcore, at least two arguments are always required:
<ul
- The
vmlinuxfile associated with the running kernel, typically found in/usr/lib/debug/lib/modules/directory.
- <="" pre="">The kernel crash dump name,
vmcore.For example:
# crash /usr/lib/debug/lib/module/vmlinux /var/crash/127.0.0.1-2007-04-30-21/:38/vmcore [...] KERNEL: /usr/lib/debug/lib/modules/2.6.9-22.EL/vmlinux DUMPFILE: /home/eteo/crash/127.0.0.1-2007-04-30-21:38/vmcore CPUS: 1 DATE: Mon Apr 30 21:38:40 2007 UPTIME: 00:04:04 LOAD AVERAGE: 0.36, 0.23, 0.08 TASKS: 36 NODENAME: localhost.localdomain RELEASE: 2.6.9-22.EL VERSION: #1 Mon Sep 19 18:20:28 EDT 2005 MACHINE: i686 (1862 Mhz) MEMORY: 1 GB PANIC: "Oops: 0002 [#1]" (check log for details) PID: 2857 COMMAND: "bash" TASK: f7b677f0 [THREAD_INFO: f7191000] CPU: 0 STATE: TASK_RUNNING (SYSRQ) crash>Setup Diskdump
Diskdump is one of the two different crash dump facilities that we shipped with Red Hat Enterprise Linux 3 and 4. This article will not cover Netdump or Kdump.
Before you beginning setting up Diskdump on your machine, do read
/usr/share/doc/diskdumputils-version/READMEto make sure that your machine has a supported storage adapter before proceeding.Assign a disk device to dump memory. It may be:
- a full disk device (for Red Hat Enterprise Linux 3 only), e.g.
/dev/sda - a partition of a disk device, e.g.
/dev/sda2 - a swap partition (for Red Hat Enterprise Linux 4 only), e.g.
/dev/sda2
Define the disk device in
/etc/sysconfig/diskdump. In this example, we will use/dev/sda2:# vi /etc/sysconfig/diskdump add the line "DEVICE=/dev/sda2"
Load the kernel module:
# tail -f /var/log/message & # modprobe diskdump Apr 30 21:29:20 kerndev kernel: disk_dump: Maximum block size: 16384 Apr 30 21:29:20 kerndev kernel: disk_dump: total blocks required: 261770 (header 3 + bitmap 8 + memory 261759)See
/proc/diskdumpafter loading diskdump kernel module:# cat /proc/diskdump # sample_rate: 8 # block_order: 2 # fallback_on_err: 1 # allow_risky_dumps: 1 # dump_level: 0 # total_blocks: 261770 #
Format the diskdump device:
# service diskdump initialformat /dev/sda2: [100.0%]
See
/proc/diskdumpafter formatting:# tail -n2 /proc/diskdump sda2 102398310 10233405
Enable Diskdump service:
# chkconfig diskdump on # service diskdump start Starting diskdump: [ OK ] # Apr 30 21:31:19 kerndev diskdump: activating succeeded
Test that Diskdump works. The following commands will crash your machine:
# echo 1 > /proc/sys/kernel/sysrq # echo c > /proc/sysrq-trigger
Make sure that you run the above two commands in console (press Ctrl + Alt + F1), so that we can see what is happening when your system crashes. You have to perform this so that you can have a
vmcorefile to follow the rest of the paper. It will be located at/var/crash.Commonly Used Crash Commands
There are many commands in
crash. It is also possible to extendcrashby adding new commands, by writing new code and compiling it into thecrashexecutable, or creating a shared object library that can be dynamically loaded by using theextendcommand. The following are some commonly usedcrashcommands that you will likely use:- help – get help
crashhas a readily available help information built into the utility, by typinghelp. Each command has its ownman-like page, which can be viewed by typinghelp command-name.crash> help * files mod runq union alias foreach mount search vm ascii fuser net set vtop bt gdb p sig waitq btop help ps struct whatis dev irq pte swap wr dis kmem ptob sym q eval list ptov sys exit log rd task extend mach repeat timer crash version: 4.0-3.3 gdb version: 6.1 For help on any command above, enter "help ". For help on input options, enter "help input". For help on output options, enter "help output".
Tip: all the
crashcommands can be piped to external programs or redirected to files:crash> log > log.txt
This will send the in-kernel log to a local file called
log.txt.crash> ps | fgrep bash | wc -l
This will count the number of
bashtasks that were running. - sys – system data
crash> sys KERNEL: /usr/lib/debug/lib/modules/2.6.9-22.EL/vmlinux DUMPFILE: /home/eteo/crash/127.0.0.1-2007-04-30-21:38/vmcore CPUS: 1 DATE: Mon Apr 30 21:38:40 2007 UPTIME: 00:04:04 LOAD AVERAGE: 0.36, 0.23, 0.08 TASKS: 36 NODENAME: localhost.localdomain RELEASE: 2.6.9-22.EL VERSION: #1 Mon Sep 19 18:20:28 EDT 2005 MACHINE: i686 (1862 Mhz) MEMORY: 1 GB PANIC: "Oops: 0002 [#1]" (check log for details)The sys messages have information of the system (e.g. kernel release, kernel version, number of CPUs, amount of memory, etc), the time of
vmcoretaken, the operating period, and the panic (e.g. oops type, panic task/PID/command, etc). - bt – backtrace
crash> bt PID: 2857 TASK: f7b677f0 CPU: 0 COMMAND: "bash" #0 [f7191e04] start_disk_dump at f89d7bb3 #1 [f7191e18] die at c010682e #2 [f7191e48] do_page_fault at c011ab00 [...] #9 [f7191fc0] system_call at c030f918 EAX: 00000004 EBX: 00000001 ECX: b7de7000 EDX: 00000002 DS: 007b ESI: 00000002 ES: 007b EDI: b7de7000 SS: 007b ESP: bfe01650 EBP: bfe01670 CS: 0073 EIP: 003297a2 ERR: 00000004 EFLAGS: 00000246 - log – dump system message buffer
crash> log [...] SysRq : Crashing the kernel by request Unable to handle kernel NULL pointer dereference at virtual address 00000000 printing eip: c0233fa7 *pde = 3e9f3067 Oops: 0002 [#1] Modules linked in: md5 ipv6 autofs4 i2c_dev i2c_core sunrpc scsi_dump diskdump dm_mirror dm_mod button battery ac yenta_socket pcmcia_core uhci_hcd ehci_hcd shpchp snd_intel8x0 snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore ipw2200 ieee80211 ieee80211_crypt tg3 floppy ext3 jbd ata_piix libata sd_mod scsi_mod CPU: 0 EIP: 0060:[] Not tainted VLI EFLAGS: 00010246 (2.6.9-22.EL) EIP is at sysrq_handle_crash+0x0/0x8 eax: 00000063 ebx: c0370db4 ecx: 00000000 edx: 00000000 esi: 00000063 edi: 00000000 ebp: 00000000 esp: f7191f60 ds: 007b es: 007b ss: 0068 Process bash (pid: 2857, threadinfo=f7191000 task=f7b677f0) Stack: c02342d8 c032dc4e c032f105 00000003 00000002 f7b6adc0 00000002 f7191fac c01a8a13 c0362740 c0168205 f7191fac b7de7000 f7b6adc0 fffffff7 b7de7000 f7191000 c01682cf f7191fac 00000000 00000000 00000000 00000001 00000002 Call Trace: [] __handle_sysrq+0x58/0xc6 [] write_sysrq_trigger+0x23/0x29 [] vfs_write+0xb6/0xe2 [] sys_write+0x3c/0x62 [] syscall_call+0x7/0xb Code: 4c 11 42 c0 05 00 00 00 c7 05 50 11 42 c0 2f cc 31 c0 c7 05 54 11 42 c0 00 00 00 00 c7 05 58 11 42 c0 00 00 00 00 e9 e5 0b f0 ff 05 00 00 00 00 00 c3 e9 e1 59 f3 ff e9 1e bc f3 ff 85 d2 89The log command dumps the kernel log buffer contents in chronological order. This is similar to what you would see when you type
dmesgon a running machine. This is useful when you want to look at the panic or oops message. An oops is triggered by some exception. It is a dump of the CPU register's state and kernel stack at that instant. From the panic message, we can find hints as to how the panic was triggered (e.g. the function or process or pid or command or address that triggered the panic), the register's information, kernel module list, whether the kernel is tainted with proprietary kernel modules loaded, and so on. Let’s walk through the panic message to see what we can learn from it. See the comments below each section within the log:crash> log [...] SysRq : Crashing the kernel by request <-- this panic is intentional Unable to handle kernel NULL pointer dereference at virtual address 00000000
This is the address to which reference was attempted.
printing eip: c0233fa7
This is the address at which the failure occurred.
*pde = 3e9f3067 Oops: 0002 [#1]
Often one oops will trigger more; only the first is reliable.
Modules linked in: md5 ipv6 autofs4 i2c_dev i2c_core sunrpc scsi_dump diskdump dm_mirror dm_mod button battery ac yenta_socket pcmcia_core uhci_hcd ehci_hcd shpchp snd_intel8x0 snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore ipw2200 ieee80211 ieee80211_crypt tg3 floppy ext3 jbd ata_piix libata sd_mod scsi_mod CPU: 0 EIP: 0060:[] Not tainted VLI
The first part is the code segment and instruction address. If tainted, it will be followed by:
G – All modules loaded have a GPL or compatible license P – Proprietary modules loaded F – Module forcibly loaded S – Oops on hardware that are not SMP capable R – Module forcibly unloaded M - Machine Check Exception (MCE) occurred etc (see Further readings section).EFLAGS: 00010246 (2.6.9-22.EL)
This line denotes program status, registers information.
<pre class="EIP is at sysrq_handle_crash+0x0/0x8
eax: 00000063 ebx: c0370db4 ecx: 00000000 edx: 00000000
esi: 00000063 edi: 00000000 ebp: 00000000 esp: f7191f60
ds: 007b es: 007b ss: 0068
Process bash (pid: 2857, threadinfo=f7191000 task=f7b677f0)
Stack: c02342d8 c032dc4e c032f105 00000003 00000002 f7b6adc0 00000002These are the stack, operations, return addresses.
f7191fac f7191000 c01682cf f7191fac 00000000 00000000 00000000 00000001 00000002Call Trace:
This is the backtrace of function calls.
[] __handle_sysrq+0x58/0xc6 [] write_sysrq_trigger+0x23/0x29 [] vfs_write+0xb6/0xe2 [] sys_write+0x3c/0x62 [] syscall_call+0x7/0xb Code: 4c 11 42 c0 05 00 00 00 c7 05 50 11 42 c0 2f cc 31 c0 c7 05 54 11 42 c0 00 00 00 00 c7 05 58 11 42 c0 00 00 00 00 e9 e5 0b f0 ff 05 00 00 00 00 00 c3 e9 e1 59 f3 ff e9 1e bc f3 ff 85 d2 89
From the line
c0233fa7, we can see that this is the address at which the failure occurred. Issuing the following command can give us more hints as to which function or source code or assembly statement in the kernel triggered that:crash> dis -lr c0233fa7 /usr/src/build/614745-i686/BUILD/kernel-2.6.9/linux- 2.6.9/drivers/char/sysrq.c: 115 0xc0233fa7 : movb $0x0,0x0- ps – display process status information
crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM 0 0 0 c0358be0 RU 0.0 0 0 [swapper] 1 0 0 f7e01770 IN 0.1 1680 684 init [...] 2380 1 0 f7ac2800 IN 0.0 1604 504 mingetty 2769 2371 0 f7ac3970 IN 0.2 5740 1636 bash 2852 1 0 f7b1a880 IN 0.2 4240 2012 sshd 2855 2852 0 f7b66680 IN 0.3 8316 2756 sshd > 2857 2855 0 f7b677f0 RU 0.2 6260 1628 bash Sometimes it is useful to know which process belongs to which parent or vice versa.pshas-cand-pto show the child and parent processes.crash> ps -p 2857 PID: 0 TASK: c0358be0 CPU: 0 COMMAND: "swapper" PID: 1 TASK: f7e01770 CPU: 0 COMMAND: "init" PID: 2852 TASK: f7b1a880 CPU: 0 COMMAND: "sshd" PID: 2855 TASK: f7b66680 CPU: 0 COMMAND: "sshd" PID: 2857 TASK: f7b677f0 CPU: 0 COMMAND: "bash"- files – open files
crash> files PID: 2857 TASK: f7b677f0 CPU: 0 COMMAND: "bash" ROOT: / CWD: /root FD FILE DENTRY INODE TYPE PATH 0 f7a6e7c0 f7790198 f7b0fdcc CHR /dev/pts/0 1 f7b6adc0 f7190130 f7b9ca4c REG /proc/sysrq-trigger 2 f7a6e7c0 f7790198 f7b0fdcc CHR /dev/pts/0 10 f7a6e7c0 f7790198 f7b0fdcc CHR /dev/pts/0 255 f7a6e7c0 f7790198 f7b0fdcc CHR /dev/pts/0 crash> files 2852 PID: 2852 TASK: f7b1a880 CPU: 0 COMMAND: "sshd" ROOT: / CWD: / FD FILE DENTRY INODE TYPE PATH 0 f7b336c0 f78001d8 f7cb1ba4 CHR /dev/null 1 f7b336c0 f78001d8 f7cb1ba4 CHR /dev/null 2 f7b336c0 f78001d8 f7cb1ba4 CHR /dev/null 3 f7b69600 f7bf5280 f7aadafc SOCK socket:/[6277]
- dev – device data
crash> help dev [...] If no argument is entered, this command dumps the contents of the chrdevs and blkdevs arrays. crash> dev CHRDEV NAME OPERATIONS 1 mem (none) 4 /dev/vc/0 (none) 4 tty (none) [...] BLKDEV NAME OPERATIONS 1 ramdisk c0376d08 2 fd (unknown) 8 sd f880e070
- files – open files
- ps – display process status information
Further reading
To learn more about this topic, check out the following reference links:
-
- More about Oops
- To find out more about Diskdump, read
/usr/share/doc/diskdumputils-version/README - Red Hat Kbase: How do I configure a Netdump Server and a Netdump Client?
- Red Hat Kbase: How do I configure kexec/kdump on Red Hat Enterprise Linux 5?
Acknowledgements
The author would like to thank Guy Streeter, Wade Mealing, and Masahiro Matsuya for reviewing and suggesting improvements for several drafts of this article.
About the author
Eugene Teo is Technical Account Manager of Red Hat’s Asia Pacific Global Support Services. Eugene received his bachelor’s degree in Computing from the National University of Singapore. In his spare time, Eugene enjoys learning new things, auditing the Linux kernel source code, and contributing kernel fixes.
- a full disk device (for Red Hat Enterprise Linux 3 only), e.g.
3058
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
