crash commands

Prerequisites

The crash utilityhas the following prerequisites:

kernelobject file:

A vmlinux kernelobject file, often referred to as the namelist inthis document, which must havebeen built with the -g Cflag so that it will contain the debug data required for symbolicdebugging.

InRHEL3 installations, the vmlinux fileassociated with the running kernel is split into two files, astripped version found in the /boot directory;which has have the operating system release string appended to it,for example, vmlinux-2.4.21-4.ELsmp.The stripped file in /boot containsa link to its associated debuginfo file, which is located inthe/usr/lib/debug/boot directory.

InRHEL4, RHEL5 and RHEL6 installations, the vmlinux fileis part of the kernel debuginfo package, and is found in therelevant /usr/lib/debug/lib/modules/<release>directory.

Ideallythe kernel object file is the same kernel object file that isassociated with the memory image. However, in circumstances wherethe vmlinux fileassociated with the crash dump or live system was not builtwith the -g flag,there are work-arounds discussed later in the Invocation section.

memoryimage:

Thismay consist of a kernel crash dump file generated from any ofthe supporteddump facilties,or live system memory accessed via /dev/mem orits replacement in RHEL4/RHEL5/RHEL6, the /dev/crash driver.If no dump file argument is issued on the crash commandline, live system memory will be used by default. When examining alive system, root privileges are required.

platformprocessor types:

The crash utilityis actively developed and tested on the x86, x86_64, ia64, ppc64,arm, s390 and s390x processors. Legacy support for the Alpha and32-bit PowerPC platforms exists, but no longer actively maintained.

Linuxkernel versions:

The crash utilityis backwards-compatible to at least Red Hat 6.0 (Linux version2.2.5-15), up to Red Hat Enterprise Linux 5 (Linux version 2.6.18+).Due to the constantly shifting sands of the upstream kernelinternals, immediate support for the latest kernel versions cannot beguaranteed. However, modifications are constantly being implementedto support changes in upstream kernel versions. The intent has alwaysbeen to make the utility independent of Linux version dependencies,building in recognition of major kernel code changes so as to adaptto new kernel versions, while maintaining backwards compatibility.

Invocationoutput

Thearguments may be entered in any order. If the file arguments are notin the current directory, absolute pathnames must be used. When indoubt, simply enter crash-h toget an explanation of the command line arguments:

#crash -h Usage: crash[-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist][dumpfile] [namelist] The[namelist] argument is a pathname to an uncompressed kernel image (avmlinux file) that has been compiled with the "-g"switch, or thathas an accessible, associated, debuginfo file. If the [dumpfile] argumentis entered, then the [namelist] argument must be entered Ifthe [namelist] argument is not entered when running on a live system,a search will be made in several typical directories for fora kernel namelist file that matches the live system. [dumpfile] The[dumpfile] argument is a pathname to a kernel memory core dump file. If the [dumpfile] argument is not entered, the session will be invokedon the live system using /dev/mem, which usually requires root privileges. [mapfile] Ifthe live system kernel, or the kernel from which the [dumpfile] wasderived, was not compiled with the -g switch, then the additional [mapfile]argument is required. The [mapfile] argument may consist ofeither the associated System.map file, or the non-debug kernel namelist. However, if the [mapfile] argument is used, then the [namelist]argument must be a kernel namelist of a similar kernel versionthat was built with the -g switch. [-S] Use"/boot/System.map" as the [mapfile]. Exampleswhen running on a live system: $crash $crash /usr/tmp/vmlinux $crash /boot/System.map vmlinux.dbg $crash -S vmlinux.dbg $crash vmlinux vmlinux.dbg Exampleswhen running on a dumpfile: $crash vmlinux vmcore $crash /boot/System.map vmlinux.dbg vmcore $crash -S vmlinux.dbg vmcore $crash vmlinux vmlinux.dbg vmcore [-h[opt]] The-h option alone displays this message. If the [opt] argument is acrash command name, the help page for that command is displayed. If thestring "input" is entered, a page describing the variouscrash commandline input options is displayed. If the string "output"is entered,a page describing command line output options is displayed. [-v] Displaythe versions of crash and gdb making up this executable. [-s] Donot display any version, GPL, or crash initialization data;proceed directlyto the "crash>" prompt. [-ifile] Executethe crash command(s) in [file] prior to accepting any user inputfrom the "crash>" prompt. [-dnum] Setcrash debug level [num]. The higher the number, the more debugdata willbe printed during crash runtime.

Giventhat all invocation arguments are in order, here is an example of asuccessful invocation on a dumpfile, running a kernel that was builtwith -g,along with a vmcore dumpfile was created by the Red Hat Netdump facility:

#crashvmlinux-2.4.20-2.1.15.entsmp vmcore crash4.0-8.11 Copyright(C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc. Copyright(C) 2004, 2005, 2006 IBM Corporation Copyright(C) 1999-2006 Hewlett-Packard Co Copyright(C) 2005, 2006 Fujitsu Limited Copyright(C) 2006, 2007 VA Linux Systems Japan K.K. Copyright(C) 2005 NEC Corporation Copyright(C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright(C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Thisprogram is free software, covered by the GNU General PublicLicense, andyou are welcome to change it and/or distribute copies of it under certainconditions. Enter "help copying" to see the conditions. Thisprogram has absolutely no warranty. Enter "help warranty"for details. GNUgdb 6.1 Copyright2004 Free Software Foundation, Inc. GDBis free software, covered by the GNU General Public License, andyou are welcometo change it and/or distribute copies of it under certainconditions. Type"show copying" to see the conditions. Thereis absolutely no warranty for GDB. Type "show warranty"for details. ThisGDB was configured as "i686-pc-linux-gnu"... KERNEL:vmlinux-2.4.20-2.1.15.entsmp DUMPFILE:vmcore CPUS:1 DATE:Wed Mar 12 10:12:56 2003 UPTIME:00:38:25 LOADAVERAGE: 1.16, 0.74, 0.30 TASKS:60 NODENAME:dhcp64-220.boston.redhat.com RELEASE:2.4.20-2.1.15.entsmp VERSION:#1 SMP Tue Mar 11 16:12:22 EST 2003 MACHINE:i686 (501 Mhz) MEMORY:128 MB PANIC:"Oops: 0002" (check log for details) PID:0 COMMAND:"swapper" TASK:c038e000 CPU:0 STATE:TASK_RUNNING (PANIC) crash>

InvocationErrors

Invocationerrors will cause the crash sessionto abort upon initialization. Typically they occur as the result ofone of the following reasons:

1. The vmlinux filecontains no debug data (i.e., was built without the -g flag),and no additional debug kernel object file name was entered on thecommand line. The error message will be of the form:

   crash:/boot/vmlinux-2.4.18-14: no debugging data available

2. The vmlinux filedoes not match the dumpfile. The error message will be of the form:

   crash:vmlinux and tmp/vmcore do not match!

3. The vmlinux filecould not be found on a live system. The error message will be ofthe form:

   crash:cannot find booted kernel -- please enter namelist argument

4. Theassociated debuginfo file cannot be found. The error message will beof the form:

   crash:/boot/vmlinux-2.4.21-4.ELsmp: no debugging data available crash:vmlinux-2.4.21-4.ELsmp.debug: debuginfo file not found

5. The crash utilitybinary does not match the vmlinux and/or vmcore arguments.The error message will be of the form:

WARNING:machine type mismatch: crashutility: X86 vmlinux:X86_64 crash:vmlinux: not a supported file format

CommandInput

Upona successful session invocation on a dump file or a live kernel,the crash> promptwill appear. Interactive crash commandsare gathered using the GNU readline library,taking advantage of its command line history mechanism, andits vi or emacs commandline editing modes. Commands may also be issued to crash froma file.

CommandLine History

Thecommand line history consists of a numbered list of previously-runcommands. The full list of commands may be viewed by entering h atany time. For example:

crash>h [1]bt -a [2]ps [3]foreach bt [4]set [5]dis -rl c0221141 crash>

Commandsin the history list may be re-run in the following manners

1. Tore-run the last commandexecuted, simply enter r or !! andthen ENTER.

2. Enter r followedby the appropriate history list number, and then ENTER.

3. Enter r followedby a uniquely-identifying set of characters from the beginning ofthe previously-entered command string, and then ENTER.

4. Recycleback through the command history list using the up-arrow anddown-arrow keys until the desired command is re-displayed, and thenENTER.

5. Recycleback through the command history list using the key-strokesappropriate for the command line editing mode being used(vi or emacs)until the desired command is re-displayed, and then ENTER.

CommandOutput

crash commandscan often be verbose, and it's helpful to control the output, as wellas to be able to scroll backwards to view previous command output.So, by default, command output that would overflow the user's displayscreen is piped to /usr/bin/less,along with a prompt line that informs the user how to scroll forward,backward, or to quit the command. For example, here is an example ofwhat a pscommandmight look like:

crash>ps PID PPID CPU TASK ST %MEM VSZ RSS COMM 0 0 0 c030a000 RU 0.0 0 0 [swapper] 1 0 0 cff98000 IN 0.2 1412 468 init 2 1 0 c1446000 IN 0.0 0 0 [keventd] 3 1 0 cfffa000 IN 0.0 0 0 [kapm-idled] 4 0 0 cfff8000 IN 0.0 0 0 [ksoftirqd_CPU0] 5 0 0 cffee000 IN 0.0 0 0 [kswapd] 6 0 0 cffec000 IN 0.0 0 0 [kreclaimd] 7 0 0 c1826000 IN 0.0 0 0 [bdflush] 8 0 0 c1824000 IN 0.0 0 0 [kupdated] 9 1 0 cff90000 IN 0.0 0 0 [mdrecoveryd] 13 1 0 cf07a000 IN 0.0 0 0 [kjournald] 89 1 0 ce804000 IN 0.0 0 0 [khubd] 184 1 0 ce4d4000 IN 0.0 0 0 [kjournald] 572 1 0 cd938000 IN 0.0 440 48 dhcpcd 637 1 0 ce4a4000 IN 0.2 1476 612 syslogd 642 1 0 cd92c000 IN 0.2 2092 432 klogd 663 1 0 ce2bc000 IN 0.2 1564 612 portmap 691 1 0 cd84a000 IN 0.3 1652 668 rpc.statd 803 1 0 cd756000 IN 0.2 1400 452 apmd 828 1 0 cd6c2000 IN 0.3 18024 684 ypbind 830 828 0 cd76e000 IN 0.3 18024 684 ypbind 831 830 0 cd71c000 IN 0.3 18024 684 ypbind

--MORE -- forward: <SPACE>, <ENTER> or j backward: bor k quit: q

Thisdefault output scrolling behavior can be turned off by entering thefollowing line in a .crashrc filelocated in either the $HOME orcurrent directories:

setscroll off

Duringruntime, the following commands (or their respective builtin aliases)can be used to turn the scrolling behavior off, and back on, again:

crash>set scroll off scroll:off crash>set scroll on scroll:on crash>alias ORIGIN ALIAS COMMAND builtin man help builtin ? help builtin quit q builtin sf set scroll off builtin sn set scroll on builtin hex set radix 16 builtin dec set radix 10 builtin g gdb builtin px p -x builtin pd p -d builtin for foreach builtin size * builtin dmesg log builtin last ps -l crash>sf scroll:off crash>sn scroll:on crash>

Alternatively,command output may be redirected to a pipe or to a file usingstandard shell redirection syntax. For examples:

crash>task | grep uid uid= 3369, euid= 3369, suid= 3369, fsuid= 3369, crash>foreach bt > bt.all crash>ps >> process.data crash>kmem -i | grep SLAB > slab.pages crash>

Whena command's output is redirected to a pipe or file, thedefault /usr/bin/less behavioris turned off for that particular command.

NumericalOutput

Thedefault numerical output radix for non-pointer values is decimal,which is most often noticed when using the builtin gdb capabilityof printing formatted data structures. During runtime, the followingcommands (or their respective builtin aliases) can be used to togglethe output radix from decimal to hexadecimal, and back again:

crash>set radix 16 outputradix: 16 (hex) crash>set scroll 10 outputradix: 10 (decimal) crash>alias ORIGIN ALIAS COMMAND builtin man help builtin ? help builtin quit q builtin sf set scroll off builtin sn set scroll on builtin hex set radix 16 builtin dec set radix 10 builtin g gdb builtin px p -x builtin pd p -d builtin for foreach builtin size * builtin dmesg log crash>hex outputradix: 16 (hex) crash>dec outputradix: 10 (decimal) crash>

Alternatively,the px or pd aliasescoerce the "print" command p,to override the current output radix. For example, here the changingvalue of jiffies ona live system is printed using the current default radix, then inhexadecimal, and lastly in decimal:

crash>p jiffies jiffies= $4 = 69821055 crash>px jiffies jiffies= $5 = 0x42963aa crash>pd jiffies jiffies= $6 = 69821656 crash>

CrashContext

Upona successful invocation of a crash session,one of the existing Linux tasks is selected as the currentcontext.It is important to be aware of the current context becauseseveral crash commandsare "context-sensitive", meaning that the command isexecuted from the view-point of the current context. Therefore, theoutput of context-sensitive commands can vary depending upon whichcontext is current.

Uponinvocation of a crash session,the selection of the current context is based upon the followingcriteria:

Ondumpfiles:

• Thetask that was running when die() wascalled.

• Thetask that was running when panic() wascalled.

• Thetask that was running when an ALT-SYSRQ-c keyboard interrupt wasreceived.

• Thetask that was running when the character "c" was echoedto /proc/sysrq-trigger.

Ona live system:

• the crash taskitself.

Thecurrent context selection is shown in the session invocation data.For example, here is a session begun on a dumpfile that was createdwhen an insmod task'sattempt to install a module resulted in an "oops"violation:

#crash tmp/vm* crash4.0-8.11 Copyright(C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc. Copyright(C) 2004, 2005, 2006 IBM Corporation Copyright(C) 1999-2006 Hewlett-Packard Co Copyright(C) 2005, 2006 Fujitsu Limited Copyright(C) 2006, 2007 VA Linux Systems Japan K.K. Copyright(C) 2005 NEC Corporation Copyright(C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright(C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Thisprogram is free software, covered by the GNU General PublicLicense, andyou are welcome to change it and/or distribute copies of it under certainconditions. Enter "help copying" to see the conditions. Thisprogram has absolutely no warranty. Enter "help warranty"for details. GNUgdb 6.1 Copyright2004 Free Software Foundation, Inc. GDBis free software, covered by the GNU General Public License, andyou are welcometo change it and/or distribute copies of it under certainconditions. Type"show copying" to see the conditions. Thereis absolutely no warranty for GDB. Type "show warranty"for details. ThisGDB was configured as "i686-pc-linux-gnu"... KERNEL:tmp/vmlinux DEBUGKERNEL: tmp/vmlinux.dbg DUMPFILE:tmp/vmcore CPUS:1 DATE:Wed Mar 27 11:02:31 2002 UPTIME:00:07:24 LOADAVERAGE: 0.43, 0.42, 0.19 TASKS:68 NODENAME:anderson.boston.redhat.com RELEASE:2.4.9-26beta.48enterprise VERSION:#1 SMP Thu Mar 21 12:33:05 EST 2002 MACHINE:i686 (501 Mhz) MEMORY:128 MB PANIC:"Oops: 0002" (check log for details) PID:1696 COMMAND:"insmod" TASK:c74de000 CPU:0 STATE:TASK_RUNNING (PANIC) crash>

Duringruntime, the current context can always be displayed by enteringthe set commandwith no arguments:

crash>set PID:1696 COMMAND:"insmod" TASK:c74de000 CPU:0 STATE:TASK_RUNNING (PANIC) crash>

Changingthe Crash Context

Thecurrent context can be changed to a new task via the set command.Either of two "handles" may be used to identify a task, thePID number, or the kernel address of the task's task_struct.For example:

crash>set 1 PID:1 COMMAND:"init" TASK:c7f98000 CPU:0 STATE:TASK_RUNNING crash>set c0a52000 PID:1503 COMMAND:"cat" TASK:c0a52000 CPU:0 STATE:TASK_INTERRUPTIBLE crash>

Alternatively,the current context can be set to the task running on a given CPUnumber, or back to the panicking task. Using the same dumpfilesession shown above, in which there is only one CPU, the originalcontext may be restored using the -c CPU-number orthe -p ("panictask") options:

crash>set -c 0 PID:1696 COMMAND:"insmod" TASK:c74de000 CPU:0 STATE:TASK_RUNNING (PANIC) crash>set -p PID:1696 COMMAND:"insmod" TASK:c74de000 CPU:0 STATE:TASK_RUNNING (PANIC) crash>

Context-SensitiveCommands

Itis important to be aware that several crash commandsare context-sensitive. For example, the files commanddisplays data about the open files of a task. If it is issued with noarguments, it displays the open files data of the current context. Inthis example, the current context happens to be PID 642,the klogd daemon:

crash>files PID:642 TASK: cd92c000 CPU: 0 COMMAND: "klogd" ROOT:/ CWD: / FD FILE DENTRY INODE TYPE PATH 0 ce06c800 ce29ec60 cd8df900 REG /proc/kmsg 1 ce06cf20 ce29ebe0 cd8df740 SOCK socket:/[858] 2 ce06c5c0 ce2423a0 ce462c80 REG /boot/System.map-2.4.9-e.3enterprise

However,if the files commandis issued with either of the two task handles as an argument, then itwill display the open files data of the specified task. In thisexample, PID 12731 is specified:

crash>files 12731 PID:12731 TASK: c8150000 CPU: 0 COMMAND: "vi" ROOT:/ CWD: /tmp FD FILE DENTRY INODE TYPE PATH 0 c988cd80 ced919a0 c87fc3c0 CHR /dev/pts/11 1 c988cd80 ced919a0 c87fc3c0 CHR /dev/pts/11 2 c988cd80 ced919a0 c87fc3c0 CHR /dev/pts/11 4 c2927ae0 c6cad8a0 cd6d5040 REG /tmp/.crontab.12730.swp 5 c2927a80 c6cad9a0 c5764ac0 REG /tmp/crontab.12730

Thistype of context-sensitive behaviour is also exhibited bythe vm, bt, sig, set, net and task commands.Unless a PID or task address is specified as an argument, the outputwill reflect data concerning the current context.

Othercommands may simply default to the current context. For example,the rd commandcan read memory from an address that is specified as a user-spaceaddress. Since the rd commanddoes not accept a PID or task address as an argument, it would benecessary to be aware that the user-space access will come from theaddress space of the current context.

BuiltinHelp

Readilyavailable help information is built into the crash utility.During a session, entering the help commandwith no argument shows the following menu:

crash>help * files mod runq union alias foreach mount search vm ascii fuser net set vtop bt gdb p sig waitq btop help ps struct whatis dev irq pte swap wr dis kmem ptob sym q eval list ptov sys exit log rd task extend mach repeat timer crashversion: 4.0-8.11 gdb version: 6.1 Forhelp on any command above, enter "help <command>". Forhelp on input options, enter "helpinput". Forhelp on output options, enter "helpoutput". crash>

Eachcommand has its own man-likehelp page, which can be viewed by clicking on the command name above.Each help page details the syntax of the command and its availableoptions, a description of the command in general, a description ofeach option, and a set of examples. During a crash session,a command's help page can be displayed by entering help followedby the command name. So, for example, to get help on how to usethe set command:

crash>help set NAME set- set a process context or internal crash variable SYNOPSIS set[pid | taskp | [-c cpu] | -p] | [crash_variable [setting]] | -v DESCRIPTION Thiscommand either sets a new context, or gets the current context for display. The context can be set by the use of: pid a process PID. taskp a hexadecimal task_struct pointer. -ccpu sets the context to the active task on a cpu (dumpfilesonly). -p sets the context to the panic task, or back to the crash task on alive system. -v display the current state of internal crash variables. Ifno argument is entered, the current context is displayed. Thecontext consistsof the PID, the task pointer, the CPU, and task state. Thiscommand may also be used to set internal crash variables. If novalue argumentis entered, the current value of the crash variable is shown. These arethe crash variables, acceptable arguments, and purpose: scroll on | off controls output scrolling. scroll less /usr/bin/less as the output scrolling program. scroll more /bin/more as the output scrolling program. scroll CRASHPAGER use CRASHPAGER environment variable as the outputscrolling program. radix 10 | 16 sets output radix to 10 or 16. refresh on | off controls internal task list refresh. print_max number set maximum number of array elements to print. console device-name sets debug console device. debug number sets crash debug level. core on | off if on, drops core when the next error message isdisplayed. hash on | off controls internal list verification. silent on | off turns off initialization messages; turns off crashprompt during input file execution. (scrollingis turned off if silent is on) edit vi | emacs set line editing mode (from .crashrc file only). namelist filename name of kernel (from .crashrc file only). dumpfile filename name of core dumpfile (from .crashrc file only). zero_excluded on | off controls whether excluded pages from a dumpfile shouldreturn zero-filled memory. Internalvariables may be set in four manners: 1.entering the set command in $HOME/.crashrc. 2.entering the set command in .crashrc in the current directory. 3.executing an input file containing the set command. 4.during runtime with this command. Duringinitialization, $HOME/.crashrc is read first, followed by the .crashrcfile in the current directory. Set commands in the .crashrc file inthe current directory override those in $HOME/.crashrc. Setcommands enteredwith this command or by runtime input file override those definedin either .crashrc file. Multiple set command arguments orargument pairsmay be entered in one command line. EXAMPLES Setthe current context to task c2fe8000: crash>set c2fe8000 PID:15917 COMMAND:"bash" TASK:c2fe8000 CPU:0 STATE:TASK_INTERRUPTIBLE Setthe context back to the panicking task: crash>set -p PID:698 COMMAND:"gen12" TASK:f9d78000 CPU:2 STATE:TASK_RUNNING (PANIC) Turnoff output scrolling: crash>set scroll off scroll:off (/usr/bin/less) Showthe current state of crash internal variables: crash>set -v scroll:on (/usr/bin/less) radix:10 (decimal) refresh:on print_max:256 console:/dev/pts/2 debug:0 core:off hash:on silent:off edit:vi namelist:vmlinux dumpfile:vmcore zero_excluded:off Showthe current context: crash>set PID:1525 COMMAND:"bash" TASK:c1ede000 CPU:0 STATE:TASK_INTERRUPTIBLE

Iffor some reason a crash sessioncannot be invoked, but help information for aparticular crash commandis desired, the same help page can be displayed from a shell commandline using the -h optiontocrash:

#crash -h ascii NAME ascii- translate a hexadecimal string to ASCII SYNOPSIS asciivalue ... DESCRIPTION Translates32-bit or 64-bit hexadecimal values to ASCII. If no argument isentered, an ASCII chart is displayed. EXAMPLES Translatethe hexadecimal value of 0x62696c2f7273752f to ASCII: crash>ascii 62696c2f7273752f 62696c2f7273752f:/usr/lib Displayan ASCII chart: crash>ascii 0 1 2 3 4 5 6 7 +------------------------------- 0| NUL DLE SP 0 @ P ' p 1| SOH DC1 ! 1 A Q a q 2| STX DC2 " 2 B R b r 3| ETX DC3 # 3 C S c s 4| EOT DC4 $ 4 D T d t 5| ENQ NAK % 5 E U e u 6| ACK SYN & 6 F V f v 7| BEL ETB ` 7 G W g w 8| BS CAN ( 8 H X h x 9| HT EM ) 9 I Y i y A| LF SUB * : J Z j z B| VT ESC + ; K [ k { C| FF FS , < L \ l | D| CR GS _ = M ] m } E| SO RS . > N ^ n ~ F| SI US / ? O - o DEL #

Lastly,help concerning command input and output can be displayed byentering helpinput or helpoutput duringruntime, or crash-h input or crash-h output froma shell command line.

TheCommand Set

Each crash commandgenerally falls into one of the following categories:

• Symbolicdisplay of kernel text or data

• Systemstate

• Utilityfunctions

• SessionControl Commands

Theremainder of this section breaks the command set into categories, andgives a short description of each command in that category. However,for complete details and examples, recall that the crash utilityhas a self-contained help page for each command; to view the fullhelp page, click on the command name next to its description below.

SymbolicDisplay of Kernel Text or Data

Thefollowing commands typically take full advantage of the powerof gdb todisplay kernel data structures symbolically.

Command		Description
struct		Displaysa formatted kernel data structure type located at a given address,or at an address referred to by a symbol; if no address isspecified, the structure definition is displayed. The output canbe narrowed down to a singular member of the structure, or todisplay the offset of every member from the beginning of thestructure. A count may be appended to display an array ofstructures. Its usage is so common that two short-cuts exist suchthat the user need not enter the "struct"command name: The"pointer-to" * commandbelow can be substituted. Ifa structure name is entered as the first token on a command line,the "struct"command is actually not necessary.
union		Sameas struct command,but used for kernel data types defined as unions instead ofstructures..
*		"Pointer-to"command which can be used in lieu of entering struct or union;the gdb modulefirst determines whether the argument is a structure or a union,and then calls the appropriate function.
p		Displays thecontents of a kernel variable; the arguments are passed onto gdb's print commandfor proper formatting. Two builtin aliases, px and pd,set the numerical output radix to hexadecimal or decimal for theprint operation, temporarily overriding the current default.
whatis		Displays allavailable symbol table information concerning a data type or adata symbol.
sym		Translates akernel symbol name to its kernel virtual address and section, or akernel virtual address to its symbol name and section. It can alsobe used to dump the complete list of kernel symbols, or to querythe symbol list for all symbols containing a given sub-string.
dis		Disassembles thetext of complete kernel function, or from a specified address fora given number of instructions, or from the beginning of afunction up to a specified address.

SystemState

Themajority of crash commandscome from the following set of "kernel-aware" commands,which delve into various kernel subsystems on a system-wide orper-task basis. The task-specific commands are context-sensitive,meaning that they act upon the current context unless a PID or taskaddress is specified as an argument.

Command		Description
bt		Arguably themost useful crash command, bt displaysa task's kernel stack back-trace, including full exception framedumps. It is context-sensitive, although the -a optionwill display the stack traces of the active task on each CPU. Thiscommand is often used within the foreach wrappercommand in order to display the back traces of all tasks with onecommand.
dev		Displays dataconcerning the character and block device assignments, I/O portusage, I/O memory usage, and PCI device data.
files		Thiscontext-sensitive command displays the task's current rootdirectory and working directories, and then for each open filedescriptor, shows: its file structaddress its dentry structaddress its inode structaddress thefile type thefile's full pathname Anotheroption acts upon a specified dentry address,showing: its inode structaddress its superblock structaddress thefile type thefile's full pathname It can be calledfrom the foreach wrappercommand.
fuser		Displays a listof tasks that reference a specified filename or inode addressas the current root or working directory, an open file descriptor,or which mmap the file.
irq		Display dataconcerning interrupt request numbers and bottom-half handling.
kmem		Thiscommand has numerous options that delve into the state of severalkernel memory subsystems: generalmemory usage, similar in scope to /proc/meminfo kmalloc slabmemory allocator, including an option that lists each slab objectand its state, verifying the slab chain displayand verification of free page lists vmalloc memoryallocator vmlist contents displayand verification of the page cache the mem_map pagelist displayNUMA data, if applicable Also, given anaddress, this command searches the symbol table, the slabsubsystem, the free list, the page_hash_table,the vmlist,and the mem_map array,displaying where it was found.
log		Dumps the kernelmessage buffer chronologically, accounting for any wrap-around.
mach		Displays machineand/or processor specific data.
mod		Displays thelist of currently-loaded kernel modules. More importantly, itloads the debug data from the module object files if they areavailable, allowing symbolic debugging capability of kernelmodules.
mount		Foreach mounted filesystem, or for just a specified filesystem,displays: its vfsmount structaddress its super_block structaddress itstype itsdevice name itsmount point Options exist todump a list of a specified filesystem's open files or dirtyinodes. Filesystems may be specified by vfsmount, super_block,or inode addresses,or by device name or mount point names.
net		Displaysvarious network-related data: displayseach configured network device's net_device address,its name, and IP address displaysthe ARP cache context-sensitivedisplay of information concerning the open sockets of a task translatesan IP address expressed as a decimal or hexadecimal value into astandard numbers-and-dots notation It can be calledfrom the foreach wrappercommand.
ps		Usefulprocess status command, in typical Linux ps commandtype output, containing: PIDnumber PPIDnumber CPUnumber taskaddress processstate percentof physical memory consumed virtualaddress size residentset size commandname Also has anoption to show a task's parental hierarchy back tothe init process,and another to show all children of a task.
pte		This commandtranslates the contents of a PTE into its physical page addressand page bit settings, or if it references a swap location, theswap device and offset.
runq		Displays list oftasks on the run queue.
sig		Acontext-sensitive command which displays a task's signalinformation, including: whetheran unblocked signal is pending thepending and blocked signals thehandler data for each signal queuedsignals, if any Other optionslist the signal number/names combination for a processor type, andtranslate the contents of a sigset_t intothe signal names whose bits are set. It can be called fromtheforeach wrappercommand.
swap		For eachconfigured swap device, this command displays the same data thatis shown by the Linux command swapon-s.
sys		Re-displaysthe same system-related data that is seenduring crash initialization: thekernel object filename thedumpfile name thenumber of CPUS thedate systemuptime systemload average thenumber of tasks thenodename thekernel release and version data theprocessor type and speed theamount of memory thepanic string Other optionsdisplay information concerning the system call table, and oneallows the root userto panic a live system.
task		Thiscontext-sensitive command displays a task'scomplete task_struct contents,or one or more members of the structure. This command is oftenused within the foreach wrappercommand in order to display task_struct datafor all tasks with one command.
timer		Displays thetimer queue entries in chronological order, listing the targetfunction names, the current value of jiffies,and the expiration time of each entry.
vm		Thispowerful, context-sensitive command displays a wealth ofinformation concerning a task's virtual memory data, including: its mm_struct address itspage directory address itsresident set size itstotal virtual memory size each vm_area_struct address,along with its start and ending virtual address, flags, andsource file if applicable. optionally,every virtual page referenced by a vm_area_struct canbe translated into its physical address, or if not resident, itsfile and offset. Other optionstranslate the flags of a vm_area_struct,or display the full contents of a task's mm_struct orof each vm_area_struct.It can be called from the foreach wrappercommand.
vtop		Thiscontext-sensitive command translates a user or kernel virtualaddress to its physical address. Also displayed are: thefull PTE translation from page directory through to the pagetable the vm_area_struct datafor user virtual addresses the mem_map pagedata associated with the physical page theswap location or file location if a user virtual page is notcurrently mapped It can be calledfrom the foreach wrappercommand.
waitq		Lists the taskslinked on a specified kernel wait queue.

UtilityFunctions

Thefollowing commands are a set of useful helper commands servingvarious purposes, some simple, others quite powerful.

Command		Description
ascii		Translates anumerical value into its ASCII components; with no arguments,displays an ASCII chart.
btop		Translates abyte value (physical address) to its page number.
eval		A simplecalculator, evaluates an expression and displays the result inhexadecimal, decimal, octal and binary, and optionally showing thebit numbers set in the result.
list		Dumps theentries of a linked list of structures. It can handle lists ofstructures that are singly-linked with simple "next"pointers, or those with embedded list_head structures.The output may be constrained to simply display the address ofeach structure in the list, or if directed, also dump eachcomplete structure, or just one member of each structure. Thegathered list entries are hashed, so a corrupted list that loopsback upon itself will be recognized.
ptob		translates apage frame number to its byte value (physical address).
ptov		Translates aphysical address into a kernel virtual address by adding theappropriate PAGE_OFFSET value.
search		Searches a rangeof user or kernel memory space for given value, with an optional"don't care" bit-mask argument.
rd		Displays aspecified amount of user virtual, kernel virtual, or physicalmemory in several formats, such as 8, 16, 32 or 64 bit values,hexadecimal or decimal, symbolically, and with ASCII translations.When reading user virtual addresses, the command iscontext-sensitive.
wr		Modifies thecontents of memory on a live system. Write permissionon /dev/mem isrequired; this command should obviously be used with great care.The write operation is constrained to one 8, 16, 32 or 64 bitlocation.

SessionControl Commands

Thefollowing commands typcally aid in the efficient running ofa crash session.

Command		Description
alias		Creates asingle-word alias for a command string. Several aliases are builtinto crash;user-defined aliases may also be defined in a .crashrc file,or during a crash sessionby entering it on the command line or reading it from an inputfile.
exit		Shuts downthe crash session(same as q).
extend		Extendthe crash commandset by dynamically loading a shared object library containing oneor more user-written commands.
foreach		Quiteoften it is helpful, or even necessary, to run thesame crash context-sensitivecommand on a number of tasks by just entering one command. Thiswrapper command sets off the execution of a given crash commandon each of a defined set of tasks, temporarily changing thecurrent context to that of the targeted task before running thecommand. The set of tasks that are issued the given command can bedefined by: oneor more PID numbers oneor more task numbers oneor more command name alluser tasks allkernel tasks theactive task on each CPU Theidentifiers above may be mixed if it makes sense, such as using acombination of PIDs, task addresses, and command names. Thecontext-sensitive commands that can be issued to the selectedtasks are: bt vm task files net set sig vtop A headercontaining the PID, task address, CPU and command name will bepre-pended before the command output for each selected task.
gdb		This commandpasses its arguments directly to gdb forprocessing. This is typically not necessary, but where ambiguitiesbetween crash and gdb commandnames exist, this will force the command to be executed by gdb.
repeat		This wrappercommand repeats a crash commandindefinitely, optionally delaying a given number of secondsbetween each command execution. Obviously this command is onlyuseful when running on a live system.
set		This primarypurpose for this command is to set the crash contextto a new task, or to display the current context. It can also beused to view or change one of a set of internal crashvariablesthat modify program behavior, such as the default output radix orscrolling behavior. It can be called from the foreach wrappercommand for viewing the context data of each task.
q		Shuts downthe crash session(same as exit).

CrashUsage: A Case Study

Thesteps taken to debug a kernel crash dump are not etched in stone, andthe crash commandsused to debug a kernel issue vary according to the problem exhibited.The section contains of a casestudythatshows how the capabilities of the crash utilitywere used to to debug a specific kernel problem. However, beforedoing so, it should be noted that the following commands aretypically the most commonly-used:

bt	Display thebacktrace of the current context, or as specified with arguments.This command is typically the first command entered after startinga dumpfile session. Since the initial context is the paniccontext, it will show the function trace leading up to the kernelpanic. bt-a will show the trace of the active taskon each CPU, since there may be an interrelationship between thepanicking task on one CPU and the running task(s) on the otherCPU(s). When bt isgiven as the argument to foreach.displays the backtraces of all tasks.
struct		Printthe contents of a data structure at a specified address. Thiscommand is so common that it is typically unnecessary to enterthe struct commandname on the command line; if the first command line argument isnot a crash or gdb command,but it is the name of a known data structure,then all the command line arguments are passed tothe struct command.So for example, the following two commands yield the same result: crash>struct vm_area_struct d3cb2600 crash>vm_area_struct d3cb2600
set	Seta new task context by PID, task address, or cpu. Sinceseveral crash commandsare context-sensitive, it's helpful to be able to change thecontext to avoid having to pass the PID or task address to thosecontext-sensitive commands in order to access the data of a taskthat is not the current context.
p	Printsthe contents of a kernel variable; since it's a gateway tothe print commandof the mbedded gdb module,it can also be used to print complex C language expressions.
rd	Readmemory, which may be either kernel virtual, user virtual, orphysical, and display it several different formats and sizes.
ps	Listsbasic task information for each process; it can also displayparent and child hierarchies.
log	Dumpthe kernel log_buf,which often contains clues leading up to a subsequent kernelcrash.
foreach	Executea crash commandon all tasks, or those specified, in the system; can be usedwith bt, vm, task, files, net, set, sig and vtop.
files	Dumpthe open file descriptor data of a task; most usefully,the file, dentry and inode structureaddresses for each open file descriptor.
vm	Dumpthe virtual memory map of a task, including the vital informationconcerning each vm_area_struct makingup a task's address space. It can also dump the physical addressof each page in the address space, or if not mapped, its locationin a file or on the swap device.

ACase Study: "kernelBUG at pipe.c:120!"

Uponbringing up a crash session,a great deal of information can be gained just by the invocationdata. Here is what what displayed in this particular case:

... KERNEL:vmlinux-2.4.9-e.10.13enterprise-g DUMPFILE:vmcore-incomplete CPUS:2 DATE:Mon Feb 17 08:20:56 2003 UPTIME:4 days, 20:04:41 LOADAVERAGE: 0.95, 1.04, 1.25 TASKS:110 NODENAME:testbox.redhat.com RELEASE:2.4.9-e.10.13enterprise VERSION:#1 SMP Mon Feb 3 12:59:26 EST 2003 MACHINE:i686 (2788 Mhz) MEMORY:6 GB PANIC:"kernel BUG at pipe.c:120!" PID:20571 COMMAND:"imp" TASK:d1566000 CPU:1 STATE:TASK_RUNNING (PANIC) crash>

Inthis case the PANIC string "kernelBUG at pipe.c:120!" pointsto the exact kernel source code line at which the panic occurred.

Then,getting a backtrace of panicking task is typically the first order ofthe day:

crash>bt PID:20571 TASK: d1566000 CPU: 1 COMMAND: "imp" #0[d1567e44] die at c010785c #1[d1567e54] do_invalid_op at c0107b2c #2[d1567f0c] error_code (via invalid_op) at c01073de EAX:0000001d EBX: ed87b2e0 ECX: c02f6064 EDX: 00005fa1 EBP:00001000 DS: 0018 ESI: f640e740 ES: 0018 EDI: 00001000 CS: 0010 EIP: c0150b6d ERR: ffffffff EFLAGS: 00010292 #3[d1567f48] pipe_read at c0150b6d #4[d1567f6c] sys_read at c01468d4 #5[d1567fc0] system_call at c01072dc EAX:00000003 EBX: 0000000a ECX: 40b4e05c EDX: 00002000 DS: 002b ESI: 00002000 ES: 002b EDI: 40b4e05c SS: 002b ESP: bffe9e88 EBP: bffe9eb8 CS: 0023 EIP: 40aaa1d4 ERR: 00000003 EFLAGS: 00000286

Thebacktrace shows that the call to die() wasgenerated by an invalid_op exception.The exception was caused by the BUG() callin the pipe_read() function:

if(count && PIPE_WAITING_WRITERS(*inode) && !(filp->f_flags& O_NONBLOCK)) { /* *We know that we are going to sleep: signal *writers synchronously that there is more *room. */ wake_up_interruptible_sync(PIPE_WAIT(*inode)); if(!PIPE_EMPTY(*inode)) BUG(); gotodo_more_read; }

Inthe code segment above, the pipe_read() codehas previously down'dthe semaphore of the inode associated with the pipe, giving itexclusive access. It had read all data in the pipe, but still neededmore to satisfy the count requested.Finding that there was a writer with more data -- and who was waitingon the semaphore -- it woke up the writer. However, after doing thewakeup, it did a sanity-check on the pipe contents, and found that itwas no longer empty -- which is theoretically impossible sinceit was still holdingthe semaphore. It appeared that the writer process wrote to the pipewhile the reader process still had exclusive access -- somehowoverriding the semaphore.

Sincethe semaphore mechanism was seemingly not working, it was firstnecessary to look at the actual semaphore structureassociated with the pipe's inode. This first required looking at thefirst argument to the pipe_read() function;the whatis commandshows that it is a structfile pointer:

crash>whatis pipe_read ssize_tpipe_read(struct file *, char *, size_t, loff_t *); crash>

Usingthe bt-f option,each frame in the backtrace is expanded to show all stack data in theframe. Looking at the expansion of the sys_read() frame,we can see that the last thing pushed on the stack beforecalling pipe_read() wasthe file pointeraddress of edf3f740:

... #3[d1567f48] pipe_read at c0150b6d [RA:c01468d6 SP: d1567f4c FP: d1567f6c SIZE: 36] d1567f4c:c026701c 00000078 fffffff2 00001000 d1567f5c:00000000 edf3f740 ffffffea 00002000 d1567f6c:c01468d6 #4[d1567f6c] sys_read at c01468d4 [RA:c01072e3 SP: d1567f70 FP: d1567fc0 SIZE: 84] d1567f70:edf3f740 40b4f05c 00002000 edf3f760 d1567f80:c03683d0 fffffffb 00000001 c0120d3b d1567f90:00000046 00000046 0000000b c0350960 d1567fa0:0000000b f639eb00 c0108e0e 00000020 d1567fb0:d1566000 00002000 40b4e05c bffe9eb8 d1567fc0:c01072e3 ...

Thetask at hand is finding the inode containing the suspect semaphorefrom the file structureaddress. The file structure's f_dentry memberpoints to its dentry structure,whose d_inode memberin turn points to the pipe's inode.The struct commandcan be used to dump the complete contents of a data structure at agiven address; by tagging the .member ontothe structure name, we can print just the member desired. Byfollowing the structure chain, the inode address can be determinedlike so:

crash>struct file.f_dentry edf3f740 f_dentry= 0xdb0ec440, crash>struct dentry.d_inode db0ec440 d_inode= 0xf640e740, crash>struct inode.i_sem f640e740 i_sem= { count= { counter= 2 }, sleepers= 0, wait= { lock= { lock= 1 }, task_list= { next= 0xf640e7ac, prev= 0xf640e7ac } } }, crash>

Thedump of the semaphore structure above showed the problem:the counter valueof 2 isillegal. It should never be greater than 1;in this case a value of 2 allows two successful down operations,i.e., giving two tasks access to the pipe at the same time.

(Asan aside, determining the inode address above could also beaccomplished by using the context-sensitive files command,which dumps the associated file, dentry and inode structureaddresses for each open file descriptor of a task. The dumped filedescriptor list would contain one with a reference tothe file structureat edf3f740,and would also show the associated inode address of f640e740.)

Beforegetting a dumpfile, this same panic had occurred several times. Itwas erroneously presumed that the problem was in the pipe-handlingcode, but it was eventually determined not to be the case. Byinstrumenting a kernel with debug code, the starting counter valueof a pipe was found to be 3.Compounding that problem was the fact that the inode slab cache isone of a few special cases that presume that the freed inode'scontents are left in a legitimate state so that they do not have tobe completely reinitialized with each subsequent reallocation. Sowhen the pipe's inode was created, it received an inode with a boguscounter value.

Confirmingthe existence of bogus inode structures in the slab cache was amulti-stepped procedure. Using the command kmem commandto access the inode slab cache, we can get the addresses of all freeand currently-allocated inodes. Since there are typically severalthousand inodes, the output is extremely verbose, but here is thebeginning of it:

crash>kmem -S inode_cache CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE c7666564inode_cache 448 11563 12339 1371 4k SLAB MEMORY TOTAL ALLOCATED FREE d1d82000 d1d82040 9 9 0 FREE/ [ALLOCATED] [d1d82040] [d1d82200] [d1d823c0] [d1d82580] [d1d82740] [d1d82900] [d1d82ac0] [d1d82c80] [d1d82e40] SLAB MEMORY TOTAL ALLOCATED FREE f4e52000 f4e52040 9 7 2 FREE/ [ALLOCATED] f4e52040 (cpu 1 cache) f4e52200 (cpu 1 cache) [f4e523c0] [f4e52580] [f4e52740] [f4e52900] [f4e52ac0] [f4e52c80] [f4e52e40] ...

Inthe truncated output above, all ofthe inode address in the slab cache are dumped; the ones currently inuse are surrounded by brackets, the free ones are not.So, for example, the inodes at addressesf4e52040 and f4e52200 arefree; the others are not. The full output was pipedto a script thatpulled out just the free inode addresses (i.e., output lines startingwith three spaces), and redirectedthem into a file.The file was modified to be a crash inputfile bymaking each extracted inode address to be the arguments ofthe struct command,using its short-cut methodthat allows the dropping of the structcommandname; therefore the input file contained hundreds of crash commandsof the form:

inode.i_semf4e52040 inode.i_semf4e52200 inode.i_semf5cdc040 inode.i_semf5cdc200 inode.i_semf5cdc3c0 inode.i_semf5cdc580 ...

Notethat the struct commandwould be used by default above, as documented in its help page; ifthe first command line argument is not a crash or gdb command,but it is thename of a known data structure, it passes the arguments tothe struct command.