Crash-utility command help 汇总(2)

NAME

bt - backtrace

SYNOPSIS
bt [-a|-c cpu(s)|-g|-r|-t|-T|-l|-e|-E|-f|-F|-o|-O|-v|-p] [-R ref] [-s [-x|d]]
[-I ip] [-S sp] [pid | task]

DESCRIPTION
Display a kernel stack backtrace. If no arguments are given, the stack
trace of the current context will be displayed.

       -a  displays the stack traces of the active task on each CPU.
           (only applicable to crash dumps)
       -A  same as -a, but also displays vector registers (S390X only).
       -p  display the stack trace of the panic task only.
           (only applicable to crash dumps)
       -c cpu  display the stack trace of the active task on one or more CPUs,
           which can be specified using the format "3", "1,8,9", "1-23",
           or "1,8,9-14". (only applicable to crash dumps)
       -g  displays the stack traces of all threads in the thread group of
           the target task; the thread group leader will be displayed first.
       -r  display raw stack data, consisting of a memory dump of the two
           pages of memory containing the task_union structure.
       -t  display all text symbols found from the last known stack location
           to the top of the stack. (helpful if the back trace fails)
       -T  display all text symbols found from just above the task_struct or
           thread_info to the top of the stack. (helpful if the back trace
           fails or the -t option starts too high in the process stack).
       -l  show file and line number of each stack trace text location.
       -e  search the stack for possible kernel and user mode exception frames.
       -E  search the IRQ stacks (x86, x86_64, arm64, and ppc64), and the
           exception stacks (x86_64) for possible exception frames; all other
           arguments except for -c will be ignored since this is not a context-
           sensitive operation.
       -f  display all stack data contained in a frame; this option can be
           used to determine the arguments passed to each function; on ia64,
           the argument register contents are dumped.
    -F[F]  similar to -f, except that the stack data is displayed symbolically
           when appropriate; if the stack data references a slab cache object,
           the name of the slab cache will be displayed in brackets; on ia64,
           the substitution is done to the argument register contents.  If -F
           is entered twice, and the stack data references a slab cache object,
           both the address and the name of the slab cache will be displayed
           in brackets.
       -v  check the kernel stack of all tasks for evidence of stack overflows.
           It does so by verifying the thread_info.task pointer, ensuring that
           the thread_info.cpu is a valid cpu number, and checking the end of 
           the stack for the STACK_END_MAGIC value.
       -o  arm64: use optional backtrace method; not supported on Linux 4.14 or
           later kernels.
           x86: use old backtrace method, permissible only on kernels that were
           compiled without the -fomit-frame_pointer.
           x86_64: use old backtrace method, which dumps potentially stale
           kernel text return addresses found on the stack.
       -O  arm64: use optional backtrace method by default; subsequent usage
           of this option toggles the backtrace method.
           x86: use old backtrace method by default, permissible only on kernels
           that were compiled without the -fomit-frame_pointer; subsequent usage
           of this option toggles the backtrace method.
           x86_64: use old backtrace method by default; subsequent usage of this
           option toggles the backtrace method.
   -R ref  display stack trace only if there is a reference to this symbol
           or text address.
       -s  display the symbol name plus its offset.
       -x  when displaying a symbol offset with the -s option, override the
           default output format with hexadecimal format.
       -d  when displaying a symbol offset with the -s option, override the
           default output format with decimal format.
    -I ip  use ip as the starting text location.
    -S sp  use sp as the starting stack frame address.
      pid  displays the stack trace(s) of this pid.
    taskp  displays the stack trace the the task referenced by this hexadecimal
           task_struct pointer.
  

Multiple pid and taskp arguments may be specified.

Note that all examples below are for x86 only. The output format will differ
for other architectures. x86 backtraces from kernels that were compiled
with the --fomit-frame-pointer CFLAG occasionally will drop stack frames,
or display a stale frame reference. When in doubt as to the accuracy of a
backtrace, the -t or -T options may help fill in the blanks.

EXAMPLES
Display the stack trace of the active task(s) when the kernel panicked:

crash> bt -a
PID: 286    TASK: c0b3a000  CPU: 0   COMMAND: "in.rlogind"
#0 [c0b3be90] crash_save_current_state at c011aed0
#1 [c0b3bea4] panic at c011367c
#2 [c0b3bee8] tulip_interrupt at c01bc820
#3 [c0b3bf08] handle_IRQ_event at c010a551
#4 [c0b3bf2c] do_8259A_IRQ at c010a319
#5 [c0b3bf3c] do_IRQ at c010a653
#6 [c0b3bfbc] ret_from_intr at c0109634
   EAX: 00000000  EBX: c0e68280  ECX: 00000000  EDX: 00000004  EBP: c0b3bfbc
   DS:  0018      ESI: 00000004  ES:  0018      EDI: c0e68284 
   CS:  0010      EIP: c012f803  ERR: ffffff09  EFLAGS: 00000246 
#7 [c0b3bfbc] sys_select at c012f803
#8 [c0b3bfc0] system_call at c0109598
   EAX: 0000008e  EBX: 00000004  ECX: bfffc9a0  EDX: 00000000 
   DS:  002b      ESI: bfffc8a0  ES:  002b      EDI: 00000000 
   SS:  002b      ESP: bfffc82c  EBP: bfffd224 
   CS:  0023      EIP: 400d032e  ERR: 0000008e  EFLAGS: 00000246  

Display the stack trace of the active task on CPU 0 and 1:

crash> bt -c 0,1
PID: 0      TASK: ffffffff81a8d020  CPU: 0   COMMAND: "swapper"
 #0 [ffff880002207e90] crash_nmi_callback at ffffffff8102fee6
 #1 [ffff880002207ea0] notifier_call_chain at ffffffff8152d525
 #2 [ffff880002207ee0] atomic_notifier_call_chain at ffffffff8152d58a
 #3 [ffff880002207ef0] notify_die at ffffffff810a155e
 #4 [ffff880002207f20] do_nmi at ffffffff8152b1eb
 #5 [ffff880002207f50] nmi at ffffffff8152aab0
    [exception RIP: native_safe_halt+0xb]
    RIP: ffffffff8103eacb  RSP: ffffffff81a01ea8  RFLAGS: 00000296
    RAX: 0000000000000000  RBX: 0000000000000000  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: 0000000000000001  RDI: ffffffff81de5228
    RBP: ffffffff81a01ea8   R8: 0000000000000000   R9: 0000000000000000
    R10: 0012099429a6bea3  R11: 0000000000000000  R12: ffffffff81c066c0
    R13: 0000000000000000  R14: ffffffffffffffff  R15: ffffffff81de1000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
--- <NMI exception stack> ---
 #6 [ffffffff81a01ea8] native_safe_halt at ffffffff8103eacb
 #7 [ffffffff81a01eb0] default_idle at ffffffff810167bd
 #8 [ffffffff81a01ed0] cpu_idle at ffffffff81009fc6

PID: 38     TASK: ffff88003eaae040  CPU: 1   COMMAND: "khungtaskd"
 #0 [ffff88003ad97ce8] machine_kexec at ffffffff81038f3b
 #1 [ffff88003ad97d48] crash_kexec at ffffffff810c5da2
 #2 [ffff88003ad97e18] panic at ffffffff8152721a
 #3 [ffff88003ad97e98] watchdog at ffffffff810e6346
 #4 [ffff88003ad97ee8] kthread at ffffffff8109af06
 #5 [ffff88003ad97f48] kernel_thread at ffffffff8100c20a

Display the stack traces of task f2814000 and PID 1592:

crash> bt f2814000 1592
PID: 1018   TASK: f2814000  CPU: 1   COMMAND: "java"
 #0 [f2815db4] schedule at c011af85
 #1 [f2815de4] __down at c010600f
 #2 [f2815e14] __down_failed at c01061b3
 #3 [f2815e24] stext_lock (via drain_cpu_caches) at c025fa55
 #4 [f2815ec8] kmem_cache_shrink_nr at c013a53e
 #5 [f2815ed8] do_try_to_free_pages at c013f402
 #6 [f2815f04] try_to_free_pages at c013f8d2
 #7 [f2815f1c] _wrapped_alloc_pages at c01406bd
 #8 [f2815f40] __alloc_pages at c014079d
 #9 [f2815f60] __get_free_pages at c014083e
#10 [f2815f68] do_fork at c011cebb
#11 [f2815fa4] sys_clone at c0105ceb
#12 [f2815fc0] system_call at c010740c
    EAX: 00000078  EBX: 00000f21  ECX: bc1ffbd8  EDX: bc1ffbe0
    DS:  002b      ESI: 00000000  ES:  002b      EDI: bc1ffd04
    SS:  002b      ESP: 0807316c  EBP: 080731bc
    CS:  0023      EIP: 4012881e  ERR: 00000078  EFLAGS: 00000296

PID: 1592   TASK: c0cec000  CPU: 3   COMMAND: "httpd"
 #0 [c0ceded4] schedule at c011af85
 #1 [c0cedf04] pipe_wait at c0153083
 #2 [c0cedf58] pipe_read at c015317f
 #3 [c0cedf7c] sys_read at c0148be6
 #4 [c0cedfc0] system_call at c010740c
    EAX: 00000003  EBX: 00000004  ECX: bffed4a3  EDX: 00000001 
    DS:  002b      ESI: 00000001  ES:  002b      EDI: bffed4a3 
    SS:  002b      ESP: bffed458  EBP: bffed488 
    CS:  0023      EIP: 4024f1d4  ERR: 00000003  EFLAGS: 00000286 

In order to examine each stack frame’s contents use the bt -f option.
From the extra frame data that is displayed, the arguments passed to each
function can be determined. Re-examining the PID 1592 trace above:

crash> bt -f 1592
PID: 1592   TASK: c0cec000  CPU: 3   COMMAND: "httpd"
 #0 [c0ceded4] schedule at c011af85
    [RA: c0153088  SP: c0ceded4  FP: c0cedf04  SIZE: 52]
    c0ceded4: c0cedf00  c0cec000  ce1a6000  00000003  
    c0cedee4: c0cec000  f26152c0  cfafc8c0  c0cec000  
    c0cedef4: ef70a0a0  c0cec000  c0cedf28  c0cedf54  
    c0cedf04: c0153088  
 #1 [c0cedf04] pipe_wait at c0153083
    [RA: c0153184  SP: c0cedf08  FP: c0cedf58  SIZE: 84]
    c0cedf08: 00000000  c0cec000  00000000  00000000  
    c0cedf18: 00000000  c0a41fa0  c011d38b  c0394120  
    c0cedf28: 00000000  c0cec000  ceeebf30  ce4adf30  
    c0cedf38: 00000000  d4b60ce0  00000000  c0cedf58  
    c0cedf48: e204f820  ef70a040  00000001  c0cedf78  
    c0cedf58: c0153184  
 #2 [c0cedf58] pipe_read at c015317f
    [RA: c0148be8  SP: c0cedf5c  FP: c0cedf7c  SIZE: 36]
    c0cedf5c: ef70a040  c0cec000  00000000  00000000  
    c0cedf6c: 00000001  f27ae680  ffffffea  c0cedfbc  
    c0cedf7c: c0148be8  
 #3 [c0cedf7c] sys_read at c0148be6
    [RA: c0107413  SP: c0cedf80  FP: c0cedfc0  SIZE: 68]
    c0cedf80: f27ae680  bffed4a3  00000001  f27ae6a0  
    c0cedf90: 40160370  24000000  4019ba28  00000000  
    c0cedfa0: 00000000  fffffffe  bffba207  fffffffe  
    c0cedfb0: c0cec000  00000001  bffed4a3  bffed488  
    c0cedfc0: c0107413  
 #4 [c0cedfc0] system_call at c010740c
    EAX: 00000003  EBX: 00000004  ECX: bffed4a3  EDX: 00000001 
    DS:  002b      ESI: 00000001  ES:  002b      EDI: bffed4a3 
    SS:  002b      ESP: bffed458  EBP: bffed488 
    CS:  0023      EIP: 4024f1d4  ERR: 00000003  EFLAGS: 00000286 
    [RA: 4024f1d4  SP: c0cedfc4  FP: c0cedffc  SIZE: 60]
    c0cedfc4: 00000004  bffed4a3  00000001  00000001  
    c0cedfd4: bffed4a3  bffed488  00000003  0000002b  
    c0cedfe4: 0000002b  00000003  4024f1d4  00000023  
    c0cedff4: 00000286  bffed458  0000002b  

Typically the arguments passed to a function will be the last values
that were pushed onto the stack by the next higher-numbered function, i.e.,
the lowest stack addresses in the frame above the called function's
stack frame.  That can be verified by disassembling the calling function.
For example, the arguments passed from sys_read() to pipe_read() above
are the file pointer, the user buffer address, the count, and a pointer
to the file structure's f_pos field.  Looking at the frame #3 data for
sys_read(), the last four items pushed onto the stack (lowest addresses)
are f27ae680, bffed4a3, 00000001, and f27ae6a0 -- which are the 4 arguments
above, in that order.  Note that the first (highest address) stack content
in frame #2 data for pipe_read() is c0148be8, which is the return address
back to sys_read(). 

Dump the text symbols found in the current context’s stack:

crash> bt -t
PID: 1357   TASK: c1aa0000  CPU: 0   COMMAND: "lockd"
      START: schedule at c01190e0
  [c1aa1f28] dput at c0157dbc
  [c1aa1f4c] schedule_timeout at c0124cd4
  [c1aa1f78] svc_recv at cb22c4d8 [sunrpc]
  [c1aa1f98] put_files_struct at c011eb21
  [c1aa1fcc] nlmclnt_proc at cb237bef [lockd]
  [c1aa1ff0] kernel_thread at c0105826
  [c1aa1ff8] nlmclnt_proc at cb237a60 [lockd]

Search the current stack for possible exception frames:

crash> bt -e
PID: 286    TASK: c0b3a000  CPU: 0   COMMAND: "in.rlogind"

 KERNEL-MODE EXCEPTION FRAME AT c0b3bf44:
   EAX: 00000000  EBX: c0e68280  ECX: 00000000  EDX: 00000004  EBP: c0b3bfbc
   DS:  0018      ESI: 00000004  ES:  0018      EDI: c0e68284 
   CS:  0010      EIP: c012f803  ERR: ffffff09  EFLAGS: 00000246 

 USER-MODE EXCEPTION FRAME AT c0b3bfc4:
   EAX: 0000008e  EBX: 00000004  ECX: bfffc9a0  EDX: 00000000 
   DS:  002b      ESI: bfffc8a0  ES:  002b      EDI: 00000000 
   SS:  002b      ESP: bfffc82c  EBP: bfffd224 
   CS:  0023      EIP: 400d032e  ERR: 0000008e  EFLAGS: 00000246 

Display the back trace from a dumpfile that resulted from the execution
of the crash utility’s “sys -panic” command:

   crash> bt
   PID: 12523  TASK: c610c000  CPU: 0   COMMAND: "crash"
    #0 [c610de64] die at c01076ec
    #1 [c610de74] do_invalid_op at c01079bc
    #2 [c610df2c] error_code (via invalid_op) at c0107256
       EAX: 0000001d  EBX: c024a4c0  ECX: c02f13c4  EDX: 000026f6  EBP: c610c000
       DS:  0018      ESI: 401de2e0  ES:  0018      EDI: c610c000
       CS:  0010      EIP: c011bbb4  ERR: ffffffff  EFLAGS: 00010296
    #3 [c610df68] panic at c011bbb4
    #4 [c610df78] do_exit at c011f1fe
    #5 [c610dfc0] system_call at c0107154
       EAX: 00000001  EBX: 00000000  ECX: 00001000  EDX: 401df154
       DS:  002b      ESI: 401de2e0  ES:  002b      EDI: 00000000
       SS:  002b      ESP: bffebf0c  EBP: bffebf38
       CS:  0023      EIP: 40163afd  ERR: 00000001  EFLAGS: 00000246
 
  Display the back trace from a dumpfile that resulted from an attempt to
  insmod the sample "crash.c" kernel module that comes as part of the
  Red Hat netdump package:

   crash> bt
   PID: 1696   TASK: c74de000  CPU: 0   COMMAND: "insmod"
    #0 [c74dfdcc] die at c01076ec
    #1 [c74dfddc] do_page_fault at c0117bbc
    #2 [c74dfee0] error_code (via page_fault) at c0107256
       EAX: 00000013  EBX: cb297000  ECX: 00000000  EDX: c5962000  EBP: c74dff28
       DS:  0018      ESI: 00000000  ES:  0018      EDI: 00000000
       CS:  0010      EIP: cb297076  ERR: ffffffff  EFLAGS: 00010282
    #3 [c74dff1c] crash_init at cb297076 [crash]
    #4 [c74dff2c] sys_init_module at c011d233
    #5 [c74dffc0] system_call at c0107154
       EAX: 00000080  EBX: 08060528  ECX: 08076450  EDX: 0000000a
       DS:  002b      ESI: 0804b305  ES:  002b      EDI: 08074ed0
       SS:  002b      ESP: bffe9a90  EBP: bffe9ac8
       CS:  0023      EIP: 4012066e  ERR: 00000080  EFLAGS: 00000246

Display the symbol name plus its offset in each frame, overriding
the current output format with hexadecimal:

crash> bt -sx
PID: 1499   TASK: ffff88006af43cc0  CPU: 2   COMMAND: "su"
 #0 [ffff8800664a1c90] machine_kexec+0x167 at ffffffff810327b7
 #1 [ffff8800664a1ce0] crash_kexec+0x60 at ffffffff810a9ec0
 #2 [ffff8800664a1db0] oops_end+0xb0 at ffffffff81504160
 #3 [ffff8800664a1dd0] general_protection+0x25 at ffffffff81503435
    [exception RIP: kmem_cache_alloc+120]
    RIP: ffffffff8113cf88  RSP: ffff8800664a1e88  RFLAGS: 00010086
    RAX: 0000000000000000  RBX: ff88006ef56840ff  RCX: ffffffff8114e9e4
    RDX: 0000000000000000  RSI: 00000000000080d0  RDI: ffffffff81796020
    RBP: ffffffff81796020   R8: ffff88000a3137a0   R9: 0000000000000000
    R10: ffff88007ac97300  R11: 0000000000000400  R12: 00000000000080d0
    R13: 0000000000000292  R14: 00000000000080d0  R15: 00000000000000c0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #4 [ffff8800664a1ed0] get_empty_filp+0x74 at ffffffff8114e9e4
 #5 [ffff8800664a1ef0] sock_alloc_fd+0x23 at ffffffff8142f553
 #6 [ffff8800664a1f10] sock_map_fd+0x23 at ffffffff8142f693
 #7 [ffff8800664a1f50] sys_socket+0x43 at ffffffff814302a3
 #8 [ffff8800664a1f80] system_call_fastpath+0x16 at ffffffff81013042
    RIP: 00007f5720b368e7  RSP: 00007fff52b629a8  RFLAGS: 00010206
    RAX: 0000000000000029  RBX: ffffffff81013042  RCX: 0000000000000000
    RDX: 0000000000000009  RSI: 0000000000000003  RDI: 0000000000000010
    RBP: 000000000066f320   R8: 0000000000000001   R9: 0000000000000000
    R10: 0000000000000000  R11: 0000000000000202  R12: ffff88007ac97300
    R13: 0000000000000000  R14: 00007f571e104a80  R15: 00007f571e305048
    ORIG_RAX: 0000000000000029  CS: 0033  SS: 002b

The following three examples show the difference in the display of
the same stack frame’s contents using -f, -F, and -FF:

crash> bf -f
...
 #4 [ffff810072b47f10] vfs_write at ffffffff800789d8
    ffff810072b47f18: ffff81007e020380 ffff81007e2c2880 
    ffff810072b47f28: 0000000000000002 fffffffffffffff7 
    ffff810072b47f38: 00002b141825d000 ffffffff80078f75 
 #5 [ffff810072b47f40] sys_write at ffffffff80078f75
...
crash> bt -F
...
 #4 [ffff810072b47f10] vfs_write at ffffffff800789d8
    ffff810072b47f18: [files_cache]    [filp]           
    ffff810072b47f28: 0000000000000002 fffffffffffffff7 
    ffff810072b47f38: 00002b141825d000 sys_write+69   
 #5 [ffff810072b47f40] sys_write at ffffffff80078f75
...
crash> bf -FF
...
 #4 [ffff810072b47f10] vfs_write at ffffffff800789d8
    ffff810072b47f18: [ffff81007e020380:files_cache] [ffff81007e2c2880:filp]
    ffff810072b47f28: 0000000000000002 fffffffffffffff7 
    ffff810072b47f38: 00002b141825d000 sys_write+69  
 #5 [ffff810072b47f40] sys_write at ffffffff80078f75
...

Check the kernel stack of all tasks for evidence of a stack overflow:

crash> bt -v
PID: 5823   TASK: ffff88102aae0040  CPU: 1   COMMAND: "flush-253:0"
possible stack overflow: thread_info.task: 102efb5adc0 != ffff88102aae0040
possible stack overflow: 40ffffffff != STACK_END_MAGIC

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

gdb - gdb command

SYNOPSIS
gdb command …

DESCRIPTION
This command passes its arguments directly to gdb for processing.
This is typically not necessary, but where ambiguities between crash and
gdb command names exist, this will force the command to be executed by gdb.

Alternatively, if “set gdb on” is entered, the session will be run in a
mode where all commands are passed directly to gdb. When running in that
mode, native crash commands may be executed by preceding them with the
“crash” directive. To restore native crash mode, enter “set gdb off”.

EXAMPLES

crash> gdb help
List of classes of commands:

aliases -- Aliases of other commands
breakpoints -- Making program stop at certain points
data -- Examining data
files -- Specifying and examining files
internals -- Maintenance commands
obscure -- Obscure features
running -- Running the program
stack -- Examining the stack
status -- Status inquiries
support -- Support facilities
tracepoints -- Tracing of program execution without stopping the program
user-defined -- User-defined commands

Type "help" followed by a class name for a list of commands in that class.
Type "help" followed by command name for full documentation.
Command name abbreviations are allowed if unambiguous.

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

net - network command

SYNOPSIS
net [[-s | -S] [-xd] [-R ref] [pid | task]] [-a] [ -n [pid | task]] [-N addr]

DESCRIPTION
Displays various network related data.

If no arguments are entered, the list of network devices, names and IP
addresses are displayed. For kernels supporting namespaces, the -n option
may be used to display the list of network devices with respect to the
network namespace of a current context or a task specified by pid or task:

    -n  the namespace of the current context.
    -n pid  a process PID.
    -n task  a hexadecimal task_struct pointer.

The -s and -S options display data with respect to the current context, but
may be appended with an argument to show the socket data with respect
to a specified task:

    -s  display open network socket/sock addresses, their family and type,
        and for INET and INET6 families, their source and destination
        addresses and ports.
    -s pid  same as above, for task with process PID pid.
    -s task  same as above, for task with hexadecimal task_struct pointer task.

    -S  displays open network socket/sock addresses followed by a dump
        of both data structures.
    -S pid  same as above, with respect to process PID.
    -S task  same as above, with respect to hexadecimal task_struct pointer.

The -R option, typically invoked from “foreach net”, and in conjunction
with the -s or -S options, searches for references to a socket address,
sock address, or a file descriptor; if found, only the referenced fd, socket
or sock data will be displayed:

-R ref  socket or sock address, or file descriptor.

Other options:

    -a  display the ARP cache.
    -N addr  translates an IPv4 address expressed as a decimal or hexadecimal
        value into a standard numbers-and-dots notation.
    -x  override default output format with hexadecimal format.
    -d  override default output format with decimal format.

EXAMPLES
Display the system’s network device list:

crash> net
   NET_DEVICE     NAME   IP ADDRESS(ES)
ffff8803741c0000  lo     127.0.0.1
fff88037059c0000  eth0   10.226.229.141
ffff8803705c0000  eth1   10.226.228.250
ffff880374ad6000  usb0   169.254.95.120

Display the network device list with respect to the network namespace
of PID 2618:

crash> net -n 2618
   NET_DEVICE     NAME   IP ADDRESS(ES)
ffff880456ee7020  lo     127.0.0.1
ffff8804516a1020  eth0   10.1.9.223

Dump the ARP cache:

crash> net -a
NEIGHBOUR      IP ADDRESS     HW TYPE   HW ADDRESS         DEVICE  STATE
f38d1b00       10.16.64.14    ETHER     00:16:3e:4b:a5:4a  eth1    STALE
f38d1080       0.0.0.0        UNKNOWN   00 00 00 00 00 00  lo      NOARP
f38d1bc0       10.16.71.254   ETHER     00:00:0c:07:ac:00  eth1    REACHABLE
f38d1200       10.16.64.21    ETHER     00:16:3e:51:d8:09  eth1    REACHABLE

Display the sockets for PID 2517, using both -s and -S output formats:

crash> net -s 2517
PID: 2517   TASK: c1598000  CPU: 1   COMMAND: "rlogin"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 3  c57375dc  c1ff1850  INET:STREAM      10.1.8.20-1023      10.1.16.62-513

crash> net -S 2517
PID: 2517   TASK: c1598000  CPU: 1   COMMAND: "rlogin"
FD   SOCKET     SOCK
 3  c57375dc  c1ff1850

struct socket {
  state = SS_CONNECTED,
  flags = 131072,
  ops = 0xc023f820,
  inode = 0xc5737540,
  fasync_list = 0x0,
  file = 0xc58892b0,
  sk = 0xc1ff1850,
  wait = 0xc14d9ed4,
  type = 1,
  passcred = 0 '\000',
  tli = 0 '\000'
}
struct sock {
  sklist_next = 0xc1ff12f0,
  sklist_prev = 0xc216bc00,
  bind_next = 0x0,
  bind_pprev = 0xc0918448,
  daddr = 1041236234,
  rcv_saddr = 336068874,
  dport = 258,
  num = 1023,
  bound_dev_if = 0,
  next = 0x0,
  pprev = 0xc0286dd4,
  state = 1 '\001',
  zapped = 0 '\000',
  sport = 65283,
  family = 2,
  reuse = 0 '\000',
  ...

Translate the rcv_saddr from above into dotted-decimal notation:

crash> net -N 1041236234
10.1.16.62

From “foreach”, find all tasks with references to socket c08ea3cc:

crash> foreach net -s -R c08ea3cc
PID: 2184   TASK: c7026000  CPU: 1   COMMAND: "klines.kss"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0

PID: 2200   TASK: c670a000  CPU: 1   COMMAND: "kpanel"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0

PID: 2201   TASK: c648a000  CPU: 1   COMMAND: "kbgndwm"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0

PID: 19294  TASK: c250a000  CPU: 0   COMMAND: "prefdm"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0

PID: 2194   TASK: c62dc000  CPU: 1   COMMAND: "kaudioserver"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0

PID: 2195   TASK: c6684000  CPU: 1   COMMAND: "maudio"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0

PID: 2196   TASK: c6b58000  CPU: 1   COMMAND: "kwmsound"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0

PID: 2197   TASK: c6696000  CPU: 0   COMMAND: "kfm"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0

PID: 2199   TASK: c65ec000  CPU: 0   COMMAND: "krootwm"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0

PID: 694    TASK: c1942000  CPU: 0   COMMAND: "prefdm"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0

PID: 698    TASK: c6a2c000  CPU: 1   COMMAND: "X"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0

PID: 2159   TASK: c4a5a000  CPU: 1   COMMAND: "kwm"
FD   SOCKET     SOCK    FAMILY:TYPE         SOURCE-PORT     DESTINATION-PORT
 5  c08ea3cc  c50d3c80  INET:STREAM        0.0.0.0-1026         0.0.0.0-0

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

set - set a process context or internal crash variable

SYNOPSIS
set [[-a] [pid | taskp] | [-c cpu] | -p] | [crash_variable [setting]] | -v

DESCRIPTION
This command either sets a new context, or gets the current context for
display. The context can be set by the use of:

  pid  a process PID.
taskp  a hexadecimal task_struct pointer.
   -a  sets the pid or task as the active task on its cpu (dumpfiles only).
   -c cpu  sets the context to the active task on a cpu (dumpfiles only).
   -p  sets the context to the panic task, or back to the crash task on
       a live system.
   -v  display the current state of internal crash variables.

If no argument is entered, the current context is displayed. The context
consists of the PID, the task pointer, the CPU, and task state. The task
state shows the bits found in both the task_struct state and exit_state
fields.

This command may also be used to set internal crash variables. If no value
argument is entered, the current value of the crash variable is shown. These
are the crash variables, acceptable arguments, and purpose:

      scroll  on | off       controls output scrolling.
      scroll  less           /usr/bin/less as the output scrolling program.
      scroll  more           /bin/more as the output scrolling program.
      scroll  CRASHPAGER     use CRASHPAGER environment variable as the
                             output scrolling program.
      radix  10 | 16         sets output radix to 10 or 16.
      refresh  on | off      controls internal task list refresh.
      print_max  number      set maximum number of array elements to print.
      print_array  on | off  if on, set gdb's printing of arrays to "pretty"
                             format, with one line per element.
      console  device-name   sets debug console device.
      debug  number          sets crash debug level.
      core  on | off         if on, drops core when the next error message
                             is displayed.
      hash  on | off         controls internal list verification.
      silent  on | off       turns off initialization messages; turns off
                             crash prompt during input file execution. 
                             (scrolling is turned off if silent is on)
      edit  vi | emacs       set line editing mode (from .crashrc file only).
      namelist  filename     name of kernel (from .crashrc file only).
      zero_excluded  on | off   
                             controls whether excluded pages, or pages that
                             are missing from an incomplete dumpfile, should
                             return zero-filled memory when read.
      null-stop  on | off    if on, gdb's printing of character arrays will
                             stop at the first NULL encountered.
      gdb  on | off          if on, the crash session will be run in a mode
                             where all commands will be passed directly to
                             gdb, and the command prompt will change to 
                             "gdb>"; when running in this mode, native crash
                             commands may be executed by preceding them with
                             the "crash" directive.
      scope  text-addr       sets the text scope for viewing the definition
                             of data structures; the "text-addr" argument
                             must be a kernel or module text address, which
                             may be expressed symbolically or as a hexadecimal
                             value.
       offline  show | hide  show or hide command output that is associated
                             with offline cpus.
       redzone  on | off     if on, CONFIG_SLUB object addresses displayed by
                             the kmem command will point to the SLAB_RED_ZONE
                             padding inserted at the beginning of the object.
       error  default | redirect | filename   
                             set the destination of error messages.
                             "default": error messages are always displayed
                             on the console; if the output of a command is
                             piped to an external command or redirected
                             to a file, the error messages are also sent
                             to the pipe or file.
                             "redirect": if the output of a command is piped
                             to an external command or redirected to a file,
                             error messages are only sent to the pipe or
                             file; otherwise they are displayed on the
                             console.
                             "filename": error messages are only sent to the
                             specified filename; they are not displayed on
                             the console and are not sent to a pipe or file.

Internal variables may be set in four manners:

1. entering the set command in $HOME/.crashrc.
2. entering the set command in .crashrc in the current directory.
3. executing an input file containing the set command.
4. during runtime with this command.

During initialization, $HOME/.crashrc is read first, followed by the
.crashrc file in the current directory. Set commands in the .crashrc file
in the current directory override those in $HOME/.crashrc. Set commands
entered with this command or by runtime input file override those
defined in either .crashrc file. Multiple set command arguments or argument
pairs may be entered in one command line.

EXAMPLES
Set the current context to task c2fe8000:

crash> set c2fe8000
     PID: 15917
 COMMAND: "bash"
    TASK: c2fe8000  
     CPU: 0
   STATE: TASK_INTERRUPTIBLE

Set the context back to the panicking task:

crash> set -p
     PID: 698
 COMMAND: "gen12"
    TASK: f9d78000
     CPU: 2
   STATE: TASK_RUNNING (PANIC)

Turn off output scrolling:

crash> set scroll off
scroll: off (/usr/bin/less)

Show the current state of crash internal variables:

crash> set -v
        scroll: on (/usr/bin/less)
         radix: 10 (decimal)
       refresh: on
     print_max: 256
   print_array: off
       console: /dev/pts/2
         debug: 0
          core: off
          hash: on
        silent: off
          edit: vi
      namelist: vmlinux
 zero_excluded: off
     null-stop: on
           gdb: off
         scope: (not set)
       offline: show
       redzone: on
         error: default

Show the current context:

crash> set
     PID: 1525
 COMMAND: "bash"
    TASK: c1ede000
     CPU: 0
   STATE: TASK_INTERRUPTIBLE

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

vm - virtual memory

SYNOPSIS
vm [-p | -P vma | -M mm | -v | -m | -x | -d | [-R reference] [pid | task]]
[-f vm_flags]

DESCRIPTION
This command displays basic virtual memory information of a context,
consisting of a pointer to its mm_struct and page dirctory, its RSS and
total virtual memory size; and a list of pointers to each vm_area_struct,
its starting and ending address, vm_flags value, and file pathname. If no
arguments are entered, the current context is used. Additionally, the -p
option translates each virtual page of each VM area to its physical address.
The -R option, typically invoked from “foreach vm”, searches for references
to a supplied number, address, or filename argument, and prints only the
essential information leading up to and including the reference.
Alternatively, the -m or -v options may be used to dump the task’s mm_struct
or all of its vm_area_structs respectively. The -p, -v, -m, -R and -f
options are all mutually exclusive.

        -p    translate each virtual page to its physical address, or if
              the page is not mapped, its swap device and offset, or
              filename and offset.
    -P vma    similar to -p, but only translate the pages belonging to the
              specified VM area of a context.
     -M mm    if the mm_struct address has been removed from the task_struct
              of an exiting task, the virtual memory data cannot be displayed.
              However, if the address can be determined from the kernel stack,
              it can be entered manually in order to try to resurrect the
              virtual memory data of the task.
        -R    reference search for references to this number or filename.
        -m    dump the mm_struct assocated with the task.
        -v    dump all of the vm_area_structs associated with the task.
        -x    override the default output format for the -m or -v options
              with hexadecimal format.
        -d    override the default output format for the -m or -v options
              with decimal format.
 -f vm_flags  translate the bits of a FLAGS (vm_flags) value.
       pid    a process PID.
      task    a hexadecimal task_struct pointer.

EXAMPLES
Display the virtual memory data of the current context:

crash> vm
PID: 30986  TASK: c0440000  CPU: 0   COMMAND: "bash"
   MM       PGD       RSS    TOTAL_VM
c303fe20  c4789000    88k      1728k
  VMA      START      END     FLAGS  FILE
c0d1f540   8048000   80ad000  1875   /bin/bash
c0d1f400   80ad000   80b3000  1873   /bin/bash
c0d1f880   80b3000   80ec000    77
c0d1f0c0  40000000  40012000   875   /lib/ld-2.1.1.so
c0d1f700  40012000  40013000   873   /lib/ld-2.1.1.so
c0d1fe00  40013000  40014000    77
c0d1f580  40014000  40016000    73
c0d1f280  4001a000  4004b000    75   /usr/lib/libncurses.so.4.2
c0d1f100  4004b000  40054000    73   /usr/lib/libncurses.so.4.2
c0d1f600  40054000  40057000    73
c0d1f9c0  40057000  40059000    75   /lib/libdl-2.1.1.so
c0d1f800  40059000  4005a000    73   /lib/libdl-2.1.1.so
c0d1fd00  4005a000  40140000    75   /lib/libc-2.1.1.so
c0d1fe40  40140000  40145000    73   /lib/libc-2.1.1.so
c0d1f780  40145000  40148000    73
c0d1f140  40148000  40150000    75   /lib/libnss_files-2.1.1.so
c0d1fa80  40150000  40151000    73   /lib/libnss_files-2.1.1.so
c0d1fb00  40151000  4015a000    75   /lib/libnss_nisplus-2.1.1.so
c5f754e0  4015a000  4015b000    73   /lib/libnss_nisplus-2.1.1.so
c0d1fec0  4015b000  4016d000    75   /lib/libnsl-2.1.1.so
c5f75460  4016d000  4016e000    73   /lib/libnsl-2.1.1.so
c5f75420  4016e000  40170000    73
c5f753e0  40170000  40178000    75   /lib/libnss_nis-2.1.1.so
c5f753a0  40178000  40179000    73   /lib/libnss_nis-2.1.1.so
c0d1f240  bfffc000  c0000000   177

Display the virtual memory data along with page translations for PID 386:

crash> vm -p 386
PID: 386    TASK: c11cc000  CPU: 0   COMMAND: "atd"
   MM       PGD       RSS    TOTAL_VM
c7e30560  c10e5000    104k     1112k
  VMA      START      END     FLAGS  FILE
c0fbe6a0   8048000   804b000  1875   /usr/sbin/atd
 VIRTUAL  PHYSICAL
 8048000  20e1000
 8049000  17c6000
 804a000  1f6f000
  VMA      START      END     FLAGS  FILE
c61e0ba0   804b000   804d000  1873   /usr/sbin/atd
 VIRTUAL  PHYSICAL
 804b000  254d000
 804c000  6a9c000
  VMA      START      END     FLAGS  FILE
c61e04e0   804d000   8050000    77   
 VIRTUAL  PHYSICAL
 804d000  219d000
 804e000  2617000
 804f000  SWAP: /dev/sda8  OFFSET: 24225
  VMA      START      END     FLAGS  FILE
c61e0720  40000000  40012000   875   /lib/ld-2.1.1.so
 VIRTUAL  PHYSICAL
40000000  FILE: /lib/ld-2.1.1.so  OFFSET: 0
40001000  FILE: /lib/ld-2.1.1.so  OFFSET: 1000
40002000  FILE: /lib/ld-2.1.1.so  OFFSET: 2000
40003000  FILE: /lib/ld-2.1.1.so  OFFSET: 3000
40004000  FILE: /lib/ld-2.1.1.so  OFFSET: 4000
40005000  FILE: /lib/ld-2.1.1.so  OFFSET: 5000
...

Although the -R option is typically invoked from “foreach vm”, it can be
executed directly. This example displays all VM areas with vm_flags of 75:

crash> vm -R 75
PID: 694    TASK: c0c76000  CPU: 1   COMMAND: "crash"
   MM       PGD      RSS    TOTAL_VM
c6c43110  c0fe9000  8932k    10720k 
  VMA       START      END   FLAGS  FILE
c322c0d0  40019000  4004a000    75  /usr/lib/libncurses.so.4.2
c67537c0  40056000  40071000    75  /lib/libm-2.1.1.so
c6753d00  40072000  40074000    75  /lib/libdl-2.1.1.so
c6753540  40075000  40081000    75  /usr/lib/libz.so.1.1.3
c6753740  40085000  4016b000    75  /lib/libc-2.1.1.so

One reason to use -R directly is to pare down the output associated with
the -p option on a task with a huge address space. This example displays
the page data associated with virtual address 40121000:

crash> vm -R 40121000
PID: 694    TASK: c0c76000  CPU: 0   COMMAND: "crash"
   MM       PGD      RSS    TOTAL_VM
c6c43110  c0fe9000  8928k    10720k 
  VMA       START      END   FLAGS  FILE
c6753740  40085000  4016b000    75  /lib/libc-2.1.1.so
VIRTUAL   PHYSICAL
40121000  FILE: /lib/libc-2.1.1.so  OFFSET: 9c000

Display the mm_struct for PID 4777:

crash> vm -m 4777
PID: 4777   TASK: c0896000  CPU: 0   COMMAND: "bash"
struct mm_struct {
  mmap = 0xc6caa1c0, 
  mmap_avl = 0x0, 
  mmap_cache = 0xc6caabc0, 
  pgd = 0xc100a000, 
  count = {
    counter = 0x1
  }, 
  map_count = 0x14, 
  mmap_sem = {
    count = {
      counter = 0x1
    }, 
    waking = 0x0, 
    wait = 0x0
  }, 
  context = 0x0, 
  start_code = 0x8048000, 
  end_code = 0x809c6f7, 
  start_data = 0x0, 
  end_data = 0x80a2090, 
  start_brk = 0x80a5420, 
  brk = 0x80b9000, 
  start_stack = 0xbffff9d0, 
  arg_start = 0xbffffad1, 
  arg_end = 0xbffffad7, 
  env_start = 0xbffffad7, 
  env_end = 0xbffffff2, 
  rss = 0xf6, 
  total_vm = 0x1a3, 
  locked_vm = 0x0, 
  def_flags = 0x0, 
  cpu_vm_mask = 0x0, 
  swap_cnt = 0x23d,
  swap_address = 0x0, 
  segments = 0x0
}

Display all of the vm_area_structs for task c47d4000:

crash> vm -v c47d4000
PID: 4971   TASK: c47d4000  CPU: 1   COMMAND: "login"
struct vm_area_struct {
  vm_mm = 0xc4b0d200, 
  vm_start = 0x8048000, 
  vm_end = 0x804d000, 
  vm_next = 0xc3e3abd0, 
  vm_page_prot = {
    pgprot = 0x25
  }, 
  vm_flags = 0x1875, 
  vm_avl_height = 0x1, 
  vm_avl_left = 0x0, 
  vm_avl_right = 0x0, 
  vm_next_share = 0x0, 
  vm_pprev_share = 0xc3e3abf0, 
  vm_ops = 0xc02392a0, 
  vm_offset = 0x0, 
  vm_file = 0xc1e23660, 
  vm_pte = 0x0
}
struct vm_area_struct {
  vm_mm = 0xc4b0d200, 
  vm_start = 0x804d000, 
  vm_end = 0x804e000, 
  vm_next = 0xc3e3a010, 
  vm_page_prot = {
    pgprot = 0x25
  }, 
  vm_flags = 0x1873, 
  vm_avl_height = 0x2, 
  vm_avl_left = 0xc3e3a810, 
  vm_avl_right = 0xc3e3a010, 
  vm_next_share = 0xc3e3a810, 
  vm_pprev_share = 0xc3699c14
  ...

Translate a FLAGS value:

crash> vm -f 3875
3875: (READ|EXEC|MAYREAD|MAYWRITE|MAYEXEC|DENYWRITE|EXECUTABLE|LOCKED)

Display the page translations of the VM area at address f5604f2c:

crash> vm -P f5604f2c
PID: 5508   TASK: f56a9570  CPU: 0   COMMAND: "crond"
  VMA       START      END    FLAGS  FILE
f5604f2c    f5b000    f67000 8000075  /lib/libnss_files-2.12.so
VIRTUAL   PHYSICAL
f5b000    3fec1000
f5c000    3d3a4000
f5d000    FILE: /lib/libnss_files-2.12.so  OFFSET: 2000
f5e000    FILE: /lib/libnss_files-2.12.so  OFFSET: 3000
f5f000    FILE: /lib/libnss_files-2.12.so  OFFSET: 4000
f60000    3fd31000
f61000    3fd32000
f62000    FILE: /lib/libnss_files-2.12.so  OFFSET: 7000
f63000    FILE: /lib/libnss_files-2.12.so  OFFSET: 8000
f64000    3ff35000
f65000    FILE: /lib/libnss_files-2.12.so  OFFSET: a000
f66000    FILE: /lib/libnss_files-2.12.so  OFFSET: b000

///////////////////////////////////////////////////////////////////////////////////////////////////
NAME

btop - bytes to page

SYNOPSIS
btop address …

DESCRIPTION
This command translates a hexadecimal address to its page number.

EXAMPLES

crash> btop 512a000
512a000: 512a

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

help - get help

SYNOPSIS
help [command | all] [-]

DESCRIPTION
When entered with no argument, a list of all currently available crash
commands is listed. If a name of a crash command is entered, a man-like
page for the command is displayed. If “all” is entered, help pages
for all commands will be displayed. If neither of the above is entered,
the argument string will be passed on to the gdb help command.

A number of internal debug, statistical, and other dumpfile related
data is available with the following options:

-a - alias data
-b - shared buffer data
-B - build data
-c - numargs cache
-d - device table
-D - dumpfile contents/statistics
-e - extension table data
-f - filesys table
-g - gdb data
-h - hash_table data
-H - hash_table data (verbose)
-k - kernel_table
-K - kernel_table (verbose)
-L - LKCD page cache environment
-M <num> machine specific
-m - machdep_table
-N - net_table
-n - dumpfile contents/statistics
-o - offset_table and size_table
-p - program_context
-r - dump registers from dumpfile header
-s - symbol table data
-t - task_table
-T - task_table plus context_array
-v - vm_table
-V - vm_table (verbose)
-x - text cache
-z - help options

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

p - print the value of an expression

SYNOPSIS
p [-x|-d][-u] [expression | symbol[:cpuspec]]

DESCRIPTION
This command passes its arguments on to gdb “print” command for evaluation.

expression  an expression to be evaluated.
    symbol  a kernel symbol.
  :cpuspec  CPU specification for a per-cpu symbol:
              :             CPU of the currently selected task.
              :a[ll]        all CPUs.
              :#[-#][,...]  CPU list(s), e.g. "1,3,5", "1-3",
                            or "1,3,5-7,10".
        -x  override default output format with hexadecimal format.
        -d  override default output format with decimal format.
        -u  the expression evaluates to a user address reference.

The default output format is decimal, but that can be changed at any time
with the two built-in aliases “hex” and “dec”. Alternatively, there
are two other built-in aliases, “px” and “pd”, which force the command
output to be displayed in hexadecimal or decimal, without changing the
default mode.

EXAMPLES
Print the contents of jiffies:

crash> p jiffies
jiffies = $6 = 166532620
crash> px jiffies
jiffies = $7 = 0x9ed174b
crash> pd jiffies
jiffies = $8 = 166533160

Print the contents of the vm_area_struct “init_mm”:

crash> p init_mm
init_mm = $5 = {
  mmap = 0xc022d540, 
  mmap_avl = 0x0, 
  mmap_cache = 0x0, 
  pgd = 0xc0101000, 
  count = {
    counter = 0x6
  }, 
  map_count = 0x1, 
  mmap_sem = {
    count = {
      counter = 0x1
    }, 
    waking = 0x0, 
    wait = 0x0
  }, 
  context = 0x0, 
  start_code = 0xc0000000, 
  end_code = 0xc022b4c8, 
  start_data = 0x0, 
  end_data = 0xc0250388, 
  start_brk = 0x0, 
  brk = 0xc02928d8, 
  start_stack = 0x0, 
  arg_start = 0x0, 
  arg_end = 0x0, 
  env_start = 0x0, 
  env_end = 0x0, 
  rss = 0x0, 
  total_vm = 0x0, 
  locked_vm = 0x0, 
  def_flags = 0x0, 
  cpu_vm_mask = 0x0, 
  swap_cnt = 0x0, 
  swap_address = 0x0, 
  segments = 0x0
}

If a per-cpu symbol is entered as a argument, its data type
and all of its per-cpu addresses are displayed:

crash> p irq_stat
PER-CPU DATA TYPE:
  irq_cpustat_t irq_stat;
PER-CPU ADDRESSES:
  [0]: ffff88021e211540
  [1]: ffff88021e251540
  [2]: ffff88021e291540
  [3]: ffff88021e2d1540

To display the contents a per-cpu symbol for CPU 1, append
a cpu-specifier:

crash> p irq_stat:1
per_cpu(irq_stat, 1) = $29 = {
  __softirq_pending = 0, 
  __nmi_count = 209034, 
  apic_timer_irqs = 597509876, 
  irq_spurious_count = 0, 
  icr_read_retry_count = 2, 
  x86_platform_ipis = 0, 
  apic_perf_irqs = 209034, 
  apic_irq_work_irqs = 0, 
  irq_resched_count = 264922233, 
  irq_call_count = 7036692, 
  irq_tlb_count = 4750442, 
  irq_thermal_count = 0, 
  irq_threshold_count = 0
}

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

sig - task signal handling

SYNOPSIS
sig [[-l] | [-s sigset]] | [-g] [pid | taskp] …

DESCRIPTION
This command displays signal-handling data of one or more tasks. Multiple
task or PID numbers may be entered; if no arguments are entered, the signal
handling data of the current context will be displayed. The default display
shows:

1.  A formatted dump of the "sig" signal_struct structure referenced by
    the task_struct.  For each defined signal, it shows the sigaction
    structure address, the signal handler, the signal sigset_t mask 
    (also expressed as a 64-bit hexadecimal value), and the flags.
2.  Whether the task has an unblocked signal pending.
3.  The contents of the "blocked" and "signal" sigset_t structures
    from the task_struct/signal_struct, both of which are represented 
    as a 64-bit hexadecimal value.
4.  For each queued signal, private and/or shared, if any, its signal
    number and associated siginfo structure address.

The -l option lists the signal numbers and their name(s). The -s option
translates a 64-bit hexadecimal value representing the contents of a
sigset_t structure into the signal names whose bits are set.

    pid  a process PID.
  taskp  a hexadecimal task_struct pointer.
     -g  displays signal information for all threads in a task's 
         thread group.
     -l  displays the defined signal numbers and names.
     -s sigset  translates a 64-bit hexadecimal value representing a sigset_t
         into a list of signal names associated with the bits set.

EXAMPLES
Dump the signal-handling data of PID 8970:

crash> sig 8970
PID: 8970   TASK: f67d8560  CPU: 1   COMMAND: "procsig"
SIGNAL_STRUCT: f6018680  COUNT: 1
 SIG SIGACTION  HANDLER       MASK       FLAGS   
 [1]  f7877684  SIG_DFL 0000000000000000 0 
 [2]  f7877698  SIG_DFL 0000000000000000 0 
...
 [8]  f7877710  SIG_DFL 0000000000000000 0 
 [9]  f7877724  SIG_DFL 0000000000000000 0 
[10]  f7877738  804867a 0000000000000000 80000000 (SA_RESETHAND)
[11]  f787774c  SIG_DFL 0000000000000000 0 
[12]  f7877760  804867f 0000000000000000 10000004 (SA_SIGINFO|SA_RESTART)
[13]  f7877774  SIG_DFL 0000000000000000 0 
...
[31]  f78778dc  SIG_DFL 0000000000000000 0 
[32]  f78778f0  SIG_DFL 0000000000000000 0 
[33]  f7877904  SIG_DFL 0000000000000000 0 
[34]  f7877918  804867f 0000000000000000 10000004 (SA_SIGINFO|SA_RESTART)
[35]  f787792c  SIG_DFL 0000000000000000 0 
[36]  f7877940  SIG_DFL 0000000000000000 0 
...
[58]  f7877af8  SIG_DFL 0000000000000000 0 
[59]  f7877b0c  SIG_DFL 0000000000000000 0 
[60]  f7877b20  SIG_DFL 0000000000000000 0 
[61]  f7877b34  SIG_DFL 0000000000000000 0 
[62]  f7877b48  SIG_DFL 0000000000000000 0 
[63]  f7877b5c  SIG_DFL 0000000000000000 0 
[64]  f7877b70  804867f 0000000000000000 10000004 (SA_SIGINFO|SA_RESTART)
SIGPENDING: no
  BLOCKED: 8000000200000800
PRIVATE_PENDING
   SIGNAL: 0000000200000800
 SIGQUEUE:  SIG  SIGINFO 
             12  f51b9c84
             34  f51b9594
SHARED_PENDING
   SIGNAL: 8000000000000800
 SIGQUEUE:  SIG  SIGINFO 
             12  f51b9188
             64  f51b9d18
             64  f51b9500

Dump the signal-handling data for all tasks in the thread group containing
PID 2578:

crash> sig -g 2578
PID: 2387   TASK: f617d020  CPU: 0   COMMAND: "slapd"
SIGNAL_STRUCT: f7dede00  COUNT: 6
SIG SIGACTION  HANDLER       MASK       FLAGS
[1]  c1f60c04   a258a7 0000000000000000 10000000 (SA_RESTART)
[2]  c1f60c18   a258a7 0000000000000000 10000000 (SA_RESTART)
[3]  c1f60c2c  SIG_DFL 0000000000000000 0
[4]  c1f60c40  SIG_DFL 0000000000000000 0
[5]  c1f60c54   a258a7 0000000000000000 10000000 (SA_RESTART)
[6]  c1f60c68  SIG_DFL 0000000000000000 0
[7]  c1f60c7c  SIG_DFL 0000000000000000 0
[8]  c1f60c90  SIG_DFL 0000000000000000 0
[9]  c1f60ca4  SIG_DFL 0000000000000000 0
[10]  c1f60cb8   a25911 0000000000000000 10000000 (SA_RESTART)
...
[64]  c1f610f0  SIG_DFL 0000000000000000 0
SHARED_PENDING
   SIGNAL: 0000000000000000
 SIGQUEUE: (empty)
 
 PID: 2387   TASK: f617d020  CPU: 0   COMMAND: "slapd"
 SIGPENDING: no
    BLOCKED: 0000000000000000
 PRIVATE_PENDING
     SIGNAL: 0000000000000000
   SIGQUEUE: (empty)

 PID: 2392   TASK: f6175aa0  CPU: 0   COMMAND: "slapd"
 SIGPENDING: no
    BLOCKED: 0000000000000000
 PRIVATE_PENDING
     SIGNAL: 0000000000000000
   SIGQUEUE: (empty)

 PID: 2523   TASK: f7cd4aa0  CPU: 1   COMMAND: "slapd"
 SIGPENDING: no
    BLOCKED: 0000000000000000
 PRIVATE_PENDING
     SIGNAL: 0000000000000000
   SIGQUEUE: (empty)

 ...

Translate the sigset_t mask value, cut-and-pasted from the signal handling
data from signals 1 and 10 above:

crash> sig -s 800A000000000201
SIGHUP SIGUSR1 SIGRTMAX-14 SIGRTMAX-12 SIGRTMAX

List the signal numbers and their names:

crash> sig -l
 [1] SIGHUP
 [2] SIGINT
 [3] SIGQUIT
 [4] SIGILL
 [5] SIGTRAP
 [6] SIGABRT/SIGIOT
 [7] SIGBUS
 [8] SIGFPE
 [9] SIGKILL
[10] SIGUSR1
[11] SIGSEGV
[12] SIGUSR2
[13] SIGPIPE
[14] SIGALRM
[15] SIGTERM
[16] SIGSTKFLT
[17] SIGCHLD/SIGCLD
[18] SIGCONT
[19] SIGSTOP
[20] SIGTSTP
[21] SIGTTIN
[22] SIGTTOU
[23] SIGURG
[24] SIGXCPU
[25] SIGXFSZ
[26] SIGVTALRM
[27] SIGPROF
[28] SIGWINCH
[29] SIGIO/SIGPOLL
[30] SIGPWR
[31] SIGSYS
[32] SIGRTMIN
[33] SIGRTMIN+1
[34] SIGRTMIN+2
[35] SIGRTMIN+3
[36] SIGRTMIN+4
[37] SIGRTMIN+5
[38] SIGRTMIN+6
[39] SIGRTMIN+7
[40] SIGRTMIN+8
[41] SIGRTMIN+9
[42] SIGRTMIN+10
[43] SIGRTMIN+11
[44] SIGRTMIN+12
[45] SIGRTMIN+13
[46] SIGRTMIN+14
[47] SIGRTMIN+15
[48] SIGRTMIN+16
[49] SIGRTMAX-15
[50] SIGRTMAX-14
[51] SIGRTMAX-13
[52] SIGRTMAX-12
[53] SIGRTMAX-11
[54] SIGRTMAX-10
[55] SIGRTMAX-9
[56] SIGRTMAX-8
[57] SIGRTMAX-7
[58] SIGRTMAX-6
[59] SIGRTMAX-5
[60] SIGRTMAX-4
[61] SIGRTMAX-3
[62] SIGRTMAX-2
[63] SIGRTMAX-1
[64] SIGRTMAX

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

vtop - virtual to physical

SYNOPSIS
vtop [-c [pid | taskp]] [-u|-k] address …

DESCRIPTION
This command translates a user or kernel virtual address to its physical
address. Also displayed is the PTE translation, the vm_area_struct data
for user virtual addresses, the mem_map page data associated with the
physical page, and the swap location or file location if the page is
not mapped. The -u and -k options specify that the address is a user
or kernel virtual address; -u and -k are not necessary on processors whose
virtual addresses self-define themselves as user or kernel. User addresses
are translated with respect to the current context unless the -c option
is used. Kernel virtual addresses are translated using the swapper_pg_dir
as the base page directory unless the -c option is used.

-u                 The address is a user virtual address; only required
                  on processors with overlapping user and kernel virtual
                  address spaces.
-k                 The address is a kernel virtual address; only required
                  on processors with overlapping user and kernel virtual
                  address spaces.
-c [pid | taskp]   Translate the virtual address from the page directory
                  of the specified PID or hexadecimal task_struct pointer.
                  However, if this command is invoked from "foreach vtop",
                  the pid or taskp argument should NOT be entered; the
                  address will be translated using the page directory of
                  each task specified by "foreach".
address            A hexadecimal user or kernel virtual address.

EXAMPLES
Translate user virtual address 80b4000:

crash> vtop 80b4000
VIRTUAL   PHYSICAL
80b4000   660f000

PAGE DIRECTORY: c37f0000
  PGD: c37f0080 => e0d067
  PMD: c37f0080 => e0d067
  PTE: c0e0d2d0 => 660f067
 PAGE: 660f000

  PTE    PHYSICAL  FLAGS
660f067   660f000  (PRESENT|RW|USER|ACCESSED|DIRTY)

  VMA      START      END      FLAGS  FILE
c773daa0   80b4000   810c000    77

  PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
c0393258   660f000         0     17000  1  uptodate

Translate kernel virtual address c806e000, first using swapper_pg_dir
as the page directory base, and secondly, using the page table base
of PID 1359:

crash> vtop c806e000
VIRTUAL   PHYSICAL
c806e000  2216000

PAGE DIRECTORY: c0101000
  PGD: c0101c80 => 94063
  PMD: c0101c80 => 94063
  PTE: c00941b8 => 2216063
 PAGE: 2216000

  PTE    PHYSICAL  FLAGS
2216063   2216000  (PRESENT|RW|ACCESSED|DIRTY)

  PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
c02e9370   2216000         0         0  1  

crash> vtop -c 1359 c806e000
VIRTUAL   PHYSICAL
c806e000  2216000

PAGE DIRECTORY: c5caf000
  PGD: c5cafc80 => 94063
  PMD: c5cafc80 => 94063
  PTE: c00941b8 => 2216063
 PAGE: 2216000

  PTE    PHYSICAL  FLAGS
2216063   2216000  (PRESENT|RW|ACCESSED|DIRTY)

  PAGE    PHYSICAL   INODE     OFFSET  CNT FLAGS
c02e9370   2216000         0         0  1  

Determine swap location of user virtual address 40104000:

crash> vtop 40104000
VIRTUAL   PHYSICAL
40104000  (not mapped)

PAGE DIRECTORY: c40d8000
  PGD: c40d8400 => 6bbe067
  PMD: c40d8400 => 6bbe067
  PTE: c6bbe410 => 58bc00  

 PTE      SWAP     OFFSET
58bc00  /dev/sda8   22716

  VMA      START      END     FLAGS  FILE
c7200ae0  40104000  40b08000    73   

SWAP: /dev/sda8  OFFSET: 22716

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

dev - device data

SYNOPSIS
dev [-i | -p | -d | -D ] [-V | -v index [file]]

DESCRIPTION
If no argument is entered, this command dumps character and block
device data.

-i  display I/O port usage; on 2.4 kernels, also display I/O memory usage.
-p  display PCI device data.
-d  display disk I/O statistics:
     TOTAL: total number of allocated in-progress I/O requests
      SYNC: I/O requests that are synchronous
     ASYNC: I/O requests that are asynchronous
      READ: I/O requests that are reads (older kernels)
     WRITE: I/O requests that are writes (older kernels)
       DRV: I/O requests that are in-flight in the device driver.
            If the device driver uses blk-mq interface, this field
            shows N/A(MQ).  If not available, this column is not shown.
-D  same as -d, but filter out disks with no in-progress I/O requests.

If the dumpfile contains device dumps:

-V  display an indexed list of all device dumps present in the vmcore,
        showing their file offset, size and name.
-v  index  select and display one device dump based upon an index value
        shown by the -V option, shown in a default human-readable format;
        alternatively, the "rd -f" option along with its various format
        options may be used to further tailor the output.

file only used with -v, copy the device dump data to a file.

EXAMPLES
Display character and block device data:

crash> dev
CHRDEV    NAME              CDEV    OPERATIONS
   1      mem             f79b83c0  memory_fops
   4      /dev/vc/0       c07bc560  console_fops
   4      tty             f7af5004  tty_fops
   4      ttyS            f7b02204  tty_fops
   5      /dev/tty        c07bc440  tty_fops
   5      /dev/console    c07bc4a0  console_fops
   5      /dev/ptmx       c07bc500  ptmx_fops
   6      lp              c5797e40  lp_fops
   7      vcs             f7b03d40  vcs_fops
  10      misc            f7f68640  misc_fops
  13      input           f79b8840  input_fops
  21      sg              f7f12840  sg_fops
  29      fb              f7f8c640  fb_fops
 128      ptm             f7b02604  tty_fops
 136      pts             f7b02404  tty_fops
 162      raw             c0693e40  raw_fops
 180      usb             f79b8bc0  usb_fops
 189      usb_device      c06a0300  usbfs_device_file_operations
 216      rfcomm          f5961a04  tty_fops
 254      pcmcia          f79b82c0  ds_fops

BLKDEV    NAME             GENDISK  OPERATIONS
   1      ramdisk         f7b23480  rd_bd_op
   8      sd              f7cab280  sd_fops
   9      md              f7829b80  md_fops
  11      sr              f75c24c0  sr_bdops
  65      sd               (none)  
  66      sd               (none)  
  67      sd               (none)  
  68      sd               (none)  
  69      sd               (none)  
  70      sd               (none)  
  71      sd               (none)  
 128      sd               (none)  
 129      sd               (none)  
 130      sd               (none)  
 131      sd               (none)  
 132      sd               (none)  
 133      sd               (none)  
 134      sd               (none)  
 135      sd               (none)  
 253      device-mapper   c57a0ac0  dm_blk_dops
 254      mdp              (none)  

Display PCI data:

crash> dev -p
PCI_DEV  BU:SL.FN CLASS: VENDOR-DEVICE
c00051c0 00:00.0  Host bridge: Intel 440BX - 82443BX Host
c0005250 00:01.0  PCI bridge: Intel 440BX - 82443BX AGP
c00052e0 00:07.0  ISA bridge: Intel 82371AB PIIX4 ISA
c0005370 00:07.1  IDE interface: Intel 82371AB PIIX4 IDE
c0005400 00:07.2  USB Controller: Intel 82371AB PIIX4 USB
c0005490 00:07.3  Bridge: Intel 82371AB PIIX4 ACPI
c0005520 00:11.0  Ethernet controller: 3Com 3C905B 100bTX
c00055b0 00:13.0  PCI bridge: DEC DC21152
c0005640 01:00.0  VGA compatible controller: NVidia [PCI_DEVICE 28]
c00056d0 02:0a.0  SCSI storage controller: Adaptec AIC-7890/1
c0005760 02:0e.0  SCSI storage controller: Adaptec AIC-7880U

Display I/O port and I/O memory usage:

crash> dev -i
RESOURCE    RANGE    NAME
c03036d4  0000-ffff  PCI IO
c0302594  0000-001f  dma1
c03025b0  0020-003f  pic1
c03025cc  0040-005f  timer
c03025e8  0060-006f  keyboard
c0302604  0080-008f  dma page reg
c0302620  00a0-00bf  pic2
c030263c  00c0-00df  dma2
c0302658  00f0-00ff  fpu
c122ff20  0170-0177  ide1
c122f240  0213-0213  isapnp read
c122ff40  02f8-02ff  serial(auto)
c122ff00  0376-0376  ide1
c03186e8  03c0-03df  vga+
c122ff60  03f8-03ff  serial(auto)
c123851c  0800-083f  Intel Corporation 82371AB PIIX4 ACPI
c1238538  0840-085f  Intel Corporation 82371AB PIIX4 ACPI
c122f220  0a79-0a79  isapnp write
c122f200  0cf8-0cff  PCI conf1
c1238858  dc00-dc7f  3Com Corporation 3c905B 100BaseTX [Cyclone]
c122fc00  dc00-dc7f  00:11.0
c12380c8  dce0-dcff  Intel Corporation 82371AB PIIX4 USB
c1238d1c  e000-efff  PCI Bus #02
c1237858  e800-e8ff  Adaptec AIC-7880U
c1237458  ec00-ecff  Adaptec AHA-2940U2/W / 7890
c1239cc8  ffa0-ffaf  Intel Corporation 82371AB PIIX4 IDE

RESOURCE        RANGE        NAME
c03036f0  00000000-ffffffff  PCI mem
c0004000  00000000-0009ffff  System RAM
c03026ac  000a0000-000bffff  Video RAM area
c03026fc  000c0000-000c7fff  Video ROM
c0302718  000c9800-000cdfff  Extension ROM
c0302734  000ce000-000ce7ff  Extension ROM
c0302750  000ce800-000cffff  Extension ROM
c03026e0  000f0000-000fffff  System ROM
c0004040  00100000-07ffdfff  System RAM
c0302674  00100000-0028682b  Kernel code
c0302690  0028682c-0031c63f  Kernel data
c0004060  07ffe000-07ffffff  reserved
c1239058  ec000000-efffffff  Intel Corporation 440BX/ZX - 82443BX/ZX Host
                             bridge
c1238d54  f1000000-f1ffffff  PCI Bus #02
c1239554  f2000000-f5ffffff  PCI Bus #01
c1237074  f4000000-f5ffffff  nVidia Corporation Riva TnT2 [NV5]
c1238d38  fa000000-fbffffff  PCI Bus #02
c1237874  faffe000-faffefff  Adaptec AIC-7880U
c127ec40  faffe000-faffefff  aic7xxx
c1237474  fafff000-faffffff  Adaptec AHA-2940U2/W / 7890
c127eec0  fafff000-faffffff  aic7xxx
c1239538  fc000000-fdffffff  PCI Bus #01
c1237058  fc000000-fcffffff  nVidia Corporation Riva TnT2 [NV5]
c1238874  fe000000-fe00007f  3Com Corporation 3c905B 100BaseTX [Cyclone]
c0004080  fec00000-fec0ffff  reserved
c00040a0  fee00000-fee0ffff  reserved
c00040c0  ffe00000-ffffffff  reserved

Display disk I/O statistics:

crash> dev -d
MAJOR GENDISK            NAME     REQUEST_QUEUE      TOTAL  READ WRITE   DRV
    2 ffff81012d8a5000   fd0      ffff81012dc053c0      12     0    12     0
   22 ffff81012dc6b000   hdc      ffff81012d8ae340       2     2     0     0
    8 ffff81012dd71000   sda      ffff81012d8af040       6     0     6     6
    8 ffff81012dc77000   sdb      ffff81012d8b5740       0     0     0     0
    8 ffff81012d8d0c00   sdc      ffff81012d8ae9c0       0     0     0     0

Display the available device dumps:

crash> dev -V
INDEX  OFFSET             SIZE             NAME
  0    0x240              33558464         cxgb4_0000:02:00.4
  1    0x2001240          33558464         cxgb4_0000:03:00.4

Extract a specified device dump to file:

crash> dev -v 0 device_dump_0.bin
DEVICE: cxgb4_0000:02:00.4
33558464 bytes copied from 0x240 to device_dump_0.bin

Format and display a device’s dump data to the screen using the “rd” command:

crash> rd -f 0x240 -32 8
240:  040b69e2 00000038 000e0001 00675fd4   .i..8........_g.
250:  00000000 21600047 00000000 00000000   ....G.`!........

Display a device’s dump data to the screen using the default format:

crash> dev -v 1
DEVICE: cxgb4_0000:03:00.4
         2001240:  00000038040b69e2 00af985c000e0001   .i..8.......\...
         2001250:  2150004700000000 0000000000000000   ....G.P!........
         2001260:  0000000000000000 0000000000000000   ................
         2001270:  0000000000000000 0002fccc00000001   ................
         2001280:  00000000000027b0 0000000000000000   .'..............
...

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

ipcs - System V IPC facilities

SYNOPSIS
ipcs [-smMq] [-n pid|task] [id | addr]

DESCRIPTION
This command provides information on the System V IPC facilities. With no
arguments, the command will display kernel usage of all three factilities.

   -s  show semaphore arrays.
   -m  show shared memory segments.
   -M  show shared memory segments with additional details.
   -q  show message queues.
   id  show the data associated with this resource ID.
 addr  show the data associated with this virtual address of a
       shmid_kernel, sem_array or msq_queue.

For kernels supporting namespaces, the -n option may be used to
display the IPC facilities with respect to the namespace of a
specified task:

   -n pid   a process PID.
   -n task  a hexadecimal task_struct pointer.

EXAMPLES
Display all IPC facilities:

crash> ipcs
SHMID_KERNEL     KEY      SHMID      UID   PERMS BYTES      NATTCH STATUS
ffff880473a28310 00000000 0          0     666   90000      1       
ffff880473a28490 00000001 32769      0     666   90000      1       
ffff880473a28250 00000002 65538      0     666   90000      1       

SEM_ARRAY        KEY      SEMID      UID   PERMS NSEMS     
ffff88047200f9d0 00000000 0          0     600   1         
ffff88046f826910 00000000 32769      0     600   1         

MSG_QUEUE        KEY      MSQID      UID   PERMS USED-BYTES   MESSAGES
ffff8100036bb8d0 000079d7 0          3369  666   16640        104
ffff8100036bb3d0 000079d8 32769      3369  666   12960        81
ffff810026d751d0 000079d9 65538      3369  666   10880        68

Display shared memory usage with detailed information:

crash> ipcs -M
SHMID_KERNEL     KEY      SHMID      UID   PERMS BYTES      NATTCH STATUS
ffff880473a28310 00000000 0          0     666   90000      1       
PAGES ALLOCATED/RESIDENT/SWAPPED: 22/1/0
INODE: ffff88047239cd98

SHMID_KERNEL     KEY      SHMID      UID   PERMS BYTES      NATTCH STATUS
ffff880473a28490 00000001 32769      0     666   90000      1       
PAGES ALLOCATED/RESIDENT/SWAPPED: 22/1/0
INODE: ffff88047239c118

SHMID_KERNEL     KEY      SHMID      UID   PERMS BYTES      NATTCH STATUS
ffff880473a28250 00000002 65538      0     666   90000      1       
PAGES ALLOCATED/RESIDENT/SWAPPED: 22/1/0
INODE: ffff880470503758

Display the shared memory data associated with shmid_kernel ffff880473a28250:

crash> ipcs -M ffff880473a28250
SHMID_KERNEL     KEY      SHMID      UID   PERMS BYTES      NATTCH STATUS
ffff880473a28250 00000002 65538      0     666   90000      1       
PAGES ALLOCATED/RESIDENT/SWAPPED: 22/1/0
INODE: ffff880470503758

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

ps - display process status information

SYNOPSIS
ps [-k|-u|-G|-y policy] [-s] [-p|-c|-t|-[l|m][-C cpu]|-a|-g|-r|-S|-A]
[pid | task | command] …

DESCRIPTION
This command displays process status for selected, or all, processes
in the system. If no arguments are entered, the process data is
is displayed for all processes. Specific processes may be selected
by using the following identifier formats:

   pid  a process PID.
  task  a hexadecimal task_struct pointer.

command a command name. If a command name is made up of letters that
are all numerical values, precede the name string with a “”.
If the command string is enclosed within “’” characters, then
the encompassed string must be a POSIX extended regular expression
that will be used to match task names.

The process list may be further restricted by the following options:

    -k  restrict the output to only kernel threads.
    -u  restrict the output to only user tasks.
    -G  display only the thread group leader in a thread group.
    -y policy  restrict the output to tasks having a specified scheduling policy
        expressed by its integer value or by its (case-insensitive) name;
        multiple policies may be entered in a comma-separated list:
          0 or NORMAL
          1 or FIFO
          2 or RR
          3 or BATCH
          4 or ISO
          5 or IDLE
          6 or DEADLINE

The process identifier types may be mixed. For each task, the following
items are displayed:

1. the process PID.
2. the parent process PID.
3. the CPU number that the task ran on last.
4. the task_struct address or the kernel stack pointer of the process.
   (see -s option below)
5. the task state (RU, IN, UN, ZO, ST, TR, DE, SW, WA, PA, ID, NE).
6. the percentage of physical memory being used by this task.
7. the virtual address size of this task in kilobytes.
8. the resident set size of this task in kilobytes.
9. the command name.

The default output shows the task_struct address of each process under a
column titled “TASK”. This can be changed to show the kernel stack
pointer under a column titled “KSTACKP”.

   -s  replace the TASK column with the KSTACKP column.

On SMP machines, the active task on each CPU will be highlighted by an
angle bracket (">") preceding its information. If the crash variable
“offline” is set to “hide”, the active task on an offline CPU will
be highlighted by a “-” preceding its information.

Alternatively, information regarding parent-child relationships,
per-task time usage data, argument/environment data, thread groups,
or resource limits may be displayed:

   -p  display the parental hierarchy of selected, or all, tasks.
   -c  display the children of selected, or all, tasks.
   -t  display the task run time, start time, and cumulative user
       and system times.
   -l  display the task's last-run timestamp value, using either the
       task_struct's last_run value, the task_struct's timestamp value
       or the task_struct's sched_entity last_arrival value, whichever
       applies, of selected, or all, tasks; the list is sorted with the
       most recently-run task (with the largest timestamp) shown first,
       followed by the task's current state.
   -m  similar to -l, but the timestamp value is translated into days,
       hours, minutes, seconds, and milliseconds since the task was
       last run on a cpu.
   -C cpus  only usable with the -l or -m options, dump the timestamp data
       in per-cpu blocks, where the cpu[s] can be specified as "1,3,5",
       "1-3", "1,3,5-7,10", "all", or "a" (shortcut for "all").
   -a  display the command line arguments and environment strings of
       selected, or all, user-mode tasks.
   -g  display tasks by thread group, of selected, or all, tasks.
   -r  display resource limits (rlimits) of selected, or all, tasks.
   -S  display a summary consisting of the number of tasks in a task state.
   -A  display only the active task on each cpu.

EXAMPLES
Show the process status of all current tasks:

crash> ps
   PID    PPID  CPU   TASK    ST  %MEM   VSZ   RSS  COMM
>     0      0   3  c024c000  RU   0.0     0     0  [swapper]
>     0      0   0  c0dce000  RU   0.0     0     0  [swapper]
      0      0   1  c0fa8000  RU   0.0     0     0  [swapper]
>     0      0   2  c009a000  RU   0.0     0     0  [swapper]
      1      0   1  c0098000  IN   0.0  1096   476  init
      2      1   1  c0090000  IN   0.0     0     0  [kflushd]
      3      1   1  c000e000  IN   0.0     0     0  [kpiod]
      4      1   3  c000c000  IN   0.0     0     0  [kswapd]
      5      1   1  c0008000  IN   0.0     0     0  [mdrecoveryd]
    253      1   2  fbc4c000  IN   0.0  1088   376  portmap
    268      1   2  fbc82000  IN   0.1  1232   504  ypbind
    274    268   2  fa984000  IN   0.1  1260   556  ypbind
    321      1   1  fabf6000  IN   0.1  1264   608  syslogd
    332      1   1  fa9be000  RU   0.1  1364   736  klogd
    346      1   2  fae88000  IN   0.0  1112   472  atd
    360      1   2  faeb2000  IN   0.1  1284   592  crond
    378      1   2  fafd6000  IN   0.1  1236   560  inetd
    392      1   0  fb710000  IN   0.1  2264  1468  named
    406      1   3  fb768000  IN   0.1  1284   560  lpd
    423      1   1  fb8ac000  IN   0.1  1128   528  rpc.statd
    434      1   2  fb75a000  IN   0.0  1072   376  rpc.rquotad
    445      1   2  fb4a4000  IN   0.0  1132   456  rpc.mountd
    460      1   1  fa938000  IN   0.0     0     0  [nfsd]
    461      1   1  faa86000  IN   0.0     0     0  [nfsd]
    462      1   0  fac48000  IN   0.0     0     0  [nfsd]
    463      1   0  fb4ca000  IN   0.0     0     0  [nfsd]
    464      1   0  fb4c8000  IN   0.0     0     0  [nfsd]
    465      1   2  fba6e000  IN   0.0     0     0  [nfsd]
    466      1   1  fba6c000  IN   0.0     0     0  [nfsd]
    467      1   2  fac04000  IN   0.0     0     0  [nfsd]
    468    461   2  fa93a000  IN   0.0     0     0  [lockd]
    469    468   2  fa93e000  IN   0.0     0     0  [rpciod]
    486      1   0  fab54000  IN   0.1  1596   880  amd
    523      1   2  fa84e000  IN   0.1  1884  1128  sendmail
    538      1   0  fa82c000  IN   0.0  1112   416  gpm
    552      1   3  fa70a000  IN   0.1  2384  1220  httpd
    556    552   3  fa776000  IN   0.1  2572  1352  httpd
    557    552   2  faba4000  IN   0.1  2572  1352  httpd
    558    552   1  fa802000  IN   0.1  2572  1352  httpd
    559    552   3  fa6ee000  IN   0.1  2572  1352  httpd
    560    552   3  fa700000  IN   0.1  2572  1352  httpd
    561    552   0  fa6f0000  IN   0.1  2572  1352  httpd
    562    552   3  fa6ea000  IN   0.1  2572  1352  httpd
    563    552   0  fa67c000  IN   0.1  2572  1352  httpd
    564    552   3  fa674000  IN   0.1  2572  1352  httpd
    565    552   3  fa66a000  IN   0.1  2572  1352  httpd
    582      1   2  fa402000  IN   0.2  2968  1916  xfs
    633      1   2  fa1ec000  IN   0.2  5512  2248  innd
    636      1   3  fa088000  IN   0.1  2536   804  actived
    676      1   0  fa840000  IN   0.0  1060   384  mingetty
    677      1   1  fa590000  IN   0.0  1060   384  mingetty
    678      1   2  fa3b8000  IN   0.0  1060   384  mingetty
    679      1   0  fa5b8000  IN   0.0  1060   384  mingetty
    680      1   1  fa3a4000  IN   0.0  1060   384  mingetty
    681      1   2  fa30a000  IN   0.0  1060   384  mingetty
    683      1   3  fa5d8000  IN   0.0  1052   280  update
    686    378   1  fa3aa000  IN   0.1  2320  1136  in.rlogind
    687    686   2  f9e52000  IN   0.1  2136  1000  login
    688    687   0  f9dec000  IN   0.1  1732   976  bash
>   700    688   1  f9d62000  RU   0.0  1048   256  gen12

Display the parental hierarchy of the “crash” process on a live system:

crash> ps -p 4249
PID: 0      TASK: c0252000  CPU: 0   COMMAND: "swapper"
PID: 1      TASK: c009a000  CPU: 1   COMMAND: "init"
PID: 632    TASK: c73b6000  CPU: 1   COMMAND: "prefdm"
PID: 637    TASK: c5a4a000  CPU: 1   COMMAND: "prefdm"
PID: 649    TASK: c179a000  CPU: 0   COMMAND: "kwm"
PID: 683    TASK: c1164000  CPU: 0   COMMAND: "kfm"
PID: 1186   TASK: c165a000  CPU: 0   COMMAND: "xterm"
PID: 1188   TASK: c705e000  CPU: 1   COMMAND: "bash"
PID: 4249   TASK: c6b9a000  CPU: 0   COMMAND: "crash"

Display all children of the “kwm” window manager:

crash> ps -c kwm
  PID: 649    TASK: c179a000  CPU: 0   COMMAND: "kwm"
  PID: 682    TASK: c2d58000  CPU: 1   COMMAND: "kwmsound"
  PID: 683    TASK: c1164000  CPU: 1   COMMAND: "kfm"
  PID: 685    TASK: c053c000  CPU: 0   COMMAND: "krootwm"
  PID: 686    TASK: c13fa000  CPU: 0   COMMAND: "kpanel"
  PID: 687    TASK: c13f0000  CPU: 1   COMMAND: "kbgndwm"

Display all threads in a firefox session:

crash> ps firefox
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
  21273  21256   6  ffff81003ec15080  IN  46.3 1138276 484364  firefox
  21276  21256   6  ffff81003f49e7e0  IN  46.3 1138276 484364  firefox
  21280  21256   0  ffff81003ec1d7e0  IN  46.3 1138276 484364  firefox
  21286  21256   6  ffff81000b0d1820  IN  46.3 1138276 484364  firefox
  21287  21256   2  ffff81000b0d10c0  IN  46.3 1138276 484364  firefox
  26975  21256   5  ffff81003b5c1820  IN  46.3 1138276 484364  firefox
  26976  21256   5  ffff810023232820  IN  46.3 1138276 484364  firefox
  26977  21256   4  ffff810021a11820  IN  46.3 1138276 484364  firefox
  26978  21256   5  ffff810003159040  IN  46.3 1138276 484364  firefox
  26979  21256   5  ffff81003a058820  IN  46.3 1138276 484364  firefox

Display only the thread group leader in the firefox session:

crash> ps -G firefox
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
  21273  21256   0  ffff81003ec15080  IN  46.3 1138276 484364  firefox

Show the time usage data for pid 10318:

crash> ps -t 10318
PID: 10318  TASK: f7b85550  CPU: 5   COMMAND: "bash"
    RUN TIME: 1 days, 01:35:32
  START TIME: 5209
       UTIME: 95
       STIME: 57

Show the process status of PID 1, task f9dec000, and all nfsd tasks:

crash> ps 1 f9dec000 nfsd
   PID    PPID  CPU   TASK    ST  %MEM   VSZ   RSS  COMM
      1      0   1  c0098000  IN   0.0  1096   476  init
    688    687   0  f9dec000  IN   0.1  1732   976  bash
    460      1   1  fa938000  IN   0.0     0     0  [nfsd]
    461      1   1  faa86000  IN   0.0     0     0  [nfsd]
    462      1   0  fac48000  IN   0.0     0     0  [nfsd]
    463      1   0  fb4ca000  IN   0.0     0     0  [nfsd]
    464      1   0  fb4c8000  IN   0.0     0     0  [nfsd]
    465      1   2  fba6e000  IN   0.0     0     0  [nfsd]
    466      1   1  fba6c000  IN   0.0     0     0  [nfsd]
    467      1   2  fac04000  IN   0.0     0     0  [nfsd]

Show all kernel threads:

crash> ps -k
   PID    PPID  CPU   TASK    ST  %MEM   VSZ   RSS  COMM
      0      0   1  c0fac000  RU   0.0     0     0  [swapper]
      0      0   0  c0252000  RU   0.0     0     0  [swapper]
      2      1   1  c0fa0000  IN   0.0     0     0  [kflushd]
      3      1   1  c03de000  IN   0.0     0     0  [kpiod]
      4      1   1  c03dc000  IN   0.0     0     0  [kswapd]
      5      1   0  c0092000  IN   0.0     0     0  [mdrecoveryd]
    336      1   0  c4a9a000  IN   0.0     0     0  [rpciod]
    337      1   0  c4830000  IN   0.0     0     0  [lockd]
    487      1   1  c4ba6000  IN   0.0     0     0  [nfsd]
    488      1   0  c18c6000  IN   0.0     0     0  [nfsd]
    489      1   0  c0cac000  IN   0.0     0     0  [nfsd]
    490      1   0  c056a000  IN   0.0     0     0  [nfsd]
    491      1   0  c0860000  IN   0.0     0     0  [nfsd]
    492      1   1  c0254000  IN   0.0     0     0  [nfsd]
    493      1   0  c0a86000  IN   0.0     0     0  [nfsd]
    494      1   0  c0968000  IN   0.0     0     0  [nfsd]

Display a summary consisting of the number of tasks in a task state:

crash> ps -S
  RU: 5
  IN: 259
  UN: 31
  ZO: 1

Display only the active task, on each cpu:

crash> ps -A
    PID    PPID  CPU       TASK        ST  %MEM    VSZ    RSS  COMM
 >    10      2   1  ffff880212969710  IN   0.0      0      0   [migration/1]
 >     0      0   3  ffff884026d43520  RU   0.0      0      0   [swapper]
 >  6582      1   2  ffff880f49c52040  RU   0.0 42202472  33368  oracle
 >  9497      1   0  ffff880549ec2ab0  RU   0.0 42314692 138664  oracle

Show all tasks sorted by their task_struct’s last_run, timestamp, or
sched_entity last_arrival timestamp value, whichever applies:

crash> ps -l
[20811245123] [IN] PID: 37    TASK: f7153030  CPU: 2  COMMAND: "events/2"
[20811229959] [IN] PID: 1756  TASK: f2a5a570  CPU: 2  COMMAND: "ntpd"
[20800696644] [IN] PID: 1456  TASK: f2b1f030  CPU: 4  COMMAND: "irqbalance"
[20617047229] [IN] PID: 2324  TASK: f57f9570  CPU: 5  COMMAND: "flush-253:0"
[20617029209] [IN] PID: 49    TASK: f7167030  CPU: 4  COMMAND: "bdi-default"
[20438025365] [IN] PID: 345   TASK: f55c7ab0  CPU: 3  COMMAND: "mpt_poll_0"
[20103026046] [IN] PID: 728   TASK: f72ba570  CPU: 3  COMMAND: "edac-poller"
[20000189409] [IN] PID: 35    TASK: f7153ab0  CPU: 0  COMMAND: "events/0"
[20000179905] [IN] PID: 48    TASK: f7167570  CPU: 0  COMMAND: "sync_supers"
[19997120354] [IN] PID: 36    TASK: f7153570  CPU: 1  COMMAND: "events/1"
[19991059209] [IN] PID: 38    TASK: f715fab0  CPU: 3  COMMAND: "events/3"
[19988091608] [IN] PID: 39    TASK: f715f570  CPU: 4  COMMAND: "events/4"
[19985076530] [IN] PID: 40    TASK: f715f030  CPU: 5  COMMAND: "events/5"
[19982019106] [IN] PID: 41    TASK: f7161ab0  CPU: 6  COMMAND: "events/6"
[19982016294] [IN] PID: 29    TASK: f7109ab0  CPU: 6  COMMAND: "ksoftirqd/6"
[19838402345] [RU] PID: 2331  TASK: f297f570  CPU: 7  COMMAND: "bash"
[19837129436] [IN] PID: 2326  TASK: f2ad5030  CPU: 6  COMMAND: "sshd"
[19289476417] [IN] PID: 1772  TASK: f5665570  CPU: 5  COMMAND: "sendmail"
...

Show the most-recently run tasks on cpu 0 using both the -l and the -m
options:

crash> ps -m -C0
CPU: 0
[ 0 00:00:00.003] [RU] PID: 1205 TASK: dee03f20 CPU: 0 COMMAND: "insmod"
[ 0 00:00:00.006] [RU] PID: 770  TASK: df9e9940 CPU: 0 COMMAND: "rsyslogd"
[ 0 00:00:00.009] [IN] PID: 603  TASK: df9bcbc0 CPU: 0 COMMAND: "udevd"
[ 0 00:00:00.010] [IN] PID: 348  TASK: df9ecbc0 CPU: 0 COMMAND: "udevd"
[ 0 00:00:00.013] [IN] PID: 934  TASK: df9171a0 CPU: 0 COMMAND: "hald"
[ 0 00:00:00.023] [IN] PID: 6    TASK: df443f20 CPU: 0 COMMAND: "events/0"
[ 0 00:00:00.029] [IN] PID: 15   TASK: df46b280 CPU: 0 COMMAND: "kblockd/0"
[ 0 00:00:00.101] [IN] PID: 1168 TASK: dee01940 CPU: 0 COMMAND: "bash"
[ 0 00:00:01.404] [IN] PID: 272  TASK: dfa48ca0 CPU: 0 COMMAND: "flush-8:0"
...

crash> ps -l -C0
CPU: 0
[137146164748] [RU] PID: 1205 TASK: dee03f20 CPU: 0 COMMAND: "insmod"
[137142534372] [RU] PID: 770  TASK: df9e9940 CPU: 0 COMMAND: "rsyslogd"
[137140168469] [IN] PID: 603  TASK: df9bcbc0 CPU: 0 COMMAND: "udevd"
[137138826427] [IN] PID: 348  TASK: df9ecbc0 CPU: 0 COMMAND: "udevd"
[137135214599] [IN] PID: 934  TASK: df9171a0 CPU: 0 COMMAND: "hald"
[137125651275] [IN] PID: 6    TASK: df443f20 CPU: 0 COMMAND: "events/0"
[137119564815] [IN] PID: 15   TASK: df46b280 CPU: 0 COMMAND: "kblockd/0"
[137047715027] [IN] PID: 1168 TASK: dee01940 CPU: 0 COMMAND: "bash"
[135744209052] [IN] PID: 272  TASK: dfa48ca0 CPU: 0 COMMAND: "flush-8:0"
... 

Show the kernel stack pointer of each user task:

crash> ps -us
   PID    PPID  CPU  KSTACKP  ST  %MEM   VSZ   RSS  COMM
      1      0   0  c009bedc  IN   0.0  1096    52  init
    239      1   0  c15e7ed8  IN   0.2  1332   224  pump
    280      1   1  c7cbdedc  IN   0.2  1092   208  portmap
    295      1   0  c7481edc  IN   0.0  1232     0  ypbind
    301    295   0  c7c7bf28  IN   0.1  1260   124  ypbind
    376      1   1  c5053f28  IN   0.0  1316    40  automount
    381      1   0  c34ddf28  IN   0.2  1316   224  automount
    391      1   1  c2777f28  IN   0.2  1316   224  automount
...

Display the argument and environment data for the automount task:

crash> ps -a automount
PID: 3948   TASK: f722ee30  CPU: 0   COMMAND: "automount"
ARG: /usr/sbin/automount --timeout=60 /net program /etc/auto.net
ENV: SELINUX_INIT=YES
     CONSOLE=/dev/console
     TERM=linux
     INIT_VERSION=sysvinit-2.85
     PATH=/sbin:/usr/sbin:/bin:/usr/bin
     LC_MESSAGES=en_US
     RUNLEVEL=3
     runlevel=3
     PWD=/
     LANG=ja_JP.UTF-8
     PREVLEVEL=N
     previous=N
     HOME=/
     SHLVL=2
     _=/usr/sbin/automount

Display the tasks in the thread group containing task c20ab0b0:

crash> ps -g c20ab0b0
PID: 6425   TASK: f72f50b0  CPU: 0   COMMAND: "firefox-bin"
  PID: 6516   TASK: f71bf1b0  CPU: 0   COMMAND: "firefox-bin"
  PID: 6518   TASK: d394b930  CPU: 0   COMMAND: "firefox-bin"
  PID: 6520   TASK: c20aa030  CPU: 0   COMMAND: "firefox-bin"
  PID: 6523   TASK: c20ab0b0  CPU: 0   COMMAND: "firefox-bin"
  PID: 6614   TASK: f1f181b0  CPU: 0   COMMAND: "firefox-bin"

Display the tasks in the thread group for each instance of the
program named “multi-thread”:

crash> ps -g multi-thread
PID: 2522   TASK: 1003f0dc7f0       CPU: 1   COMMAND: "multi-thread"
  PID: 2523   TASK: 10037b13030       CPU: 1   COMMAND: "multi-thread"
  PID: 2524   TASK: 1003e064030       CPU: 1   COMMAND: "multi-thread"
  PID: 2525   TASK: 1003e13a7f0       CPU: 1   COMMAND: "multi-thread"

PID: 2526   TASK: 1002f82b7f0       CPU: 1   COMMAND: "multi-thread"
  PID: 2527   TASK: 1003e1737f0       CPU: 1   COMMAND: "multi-thread"
  PID: 2528   TASK: 10035b4b7f0       CPU: 1   COMMAND: "multi-thread"
  PID: 2529   TASK: 1003f0c37f0       CPU: 1   COMMAND: "multi-thread"
  PID: 2530   TASK: 10035597030       CPU: 1   COMMAND: "multi-thread"
  PID: 2531   TASK: 100184be7f0       CPU: 1   COMMAND: "multi-thread"

Display the resource limits of “bash” task 13896:

crash> ps -r 13896
PID: 13896  TASK: cf402000  CPU: 0   COMMAND: "bash"
   RLIMIT     CURRENT       MAXIMUM
      CPU   (unlimited)   (unlimited)
    FSIZE   (unlimited)   (unlimited)
     DATA   (unlimited)   (unlimited)
    STACK    10485760     (unlimited)
     CORE   (unlimited)   (unlimited)
      RSS   (unlimited)   (unlimited)
    NPROC      4091          4091
   NOFILE      1024          1024
  MEMLOCK      4096          4096
       AS   (unlimited)   (unlimited)
    LOCKS   (unlimited)   (unlimited)

Search for task names matching a POSIX regular expression:

 crash> ps 'migration*'
    PID    PPID  CPU       TASK        ST  %MEM    VSZ    RSS  COMM
       8      2   0  ffff8802128a2e20  IN   0.0      0      0  [migration/0]
      10      2   1  ffff880212969710  IN   0.0      0      0  [migration/1]
      15      2   2  ffff880212989710  IN   0.0      0      0  [migration/2]
      20      2   3  ffff8802129a9710  IN   0.0      0      0  [migration/3]

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

struct - structure contents

SYNOPSIS
struct struct_name[.member[,member]][-o][-l offset][-rfuxdp]
[address | symbol][:cpuspec] [count | -c count]

DESCRIPTION
This command displays either a structure definition, or a formatted display
of the contents of a structure at a specified address. When no address is
specified, the structure definition is shown along with the structure size.
A structure member may be appended to the structure name in order to limit
the scope of the data displayed to that particular member; when no address
is specified, the member’s offset and definition are shown.

struct_name  name of a C-code structure used by the kernel.
    .member  name of a structure member; to display multiple members of a
             structure, use a comma-separated list of members.  If any
             member contains an embedded structure, or the member is an 
             array, the output may be restricted to just the embedded 
             structure or an array element by expressing the member argument
             as "member.member" or "member[index]"; embedded member
             specifications may extend beyond one level deep, by expressing
             the member argument as "member.member.member...".
         -o  show member offsets when displaying structure definitions; 
             if used with an address or symbol argument, each member will
             be preceded by its virtual address.
  -l offset  if the address argument is a pointer to a structure member that
             is contained by the target data structure, typically a pointer
             to an embedded list_head, the offset to the embedded member may
             be entered in either of the following manners:
               1. in "structure.member" format.
               2. a number of bytes. 
         -r  raw dump of structure data.
         -f  address argument is a dumpfile offset.
         -u  address argument is a user virtual address in the current
             context.
         -x  override default output format with hexadecimal format.
         -d  override default output format with decimal format.
         -p  if a structure member is a pointer value, show the member's
             data type on the output line; and on the subsequent line(s),
             dereference the pointer, display the pointer target's symbol
             value in brackets if appropriate, and if possible, display the
             target data; requires an address argument.
    address  hexadecimal address of a structure; if the address points
             to an embedded list_head structure contained within the
             target data structure, then the "-l" option must be used.
     symbol  symbolic reference to the address of a structure.
   :cpuspec  CPU specification for a per-cpu address or symbol:
               :             CPU of the currently selected task.
               :a[ll]        all CPUs.
               :#[-#][,...]  CPU list(s), e.g. "1,3,5", "1-3",
                             or "1,3,5-7,10".
      count  count of structures to dump from an array of structures;
             if used, this must be the last argument entered.
   -c count  "-c" is only required if "count" is not the last argument
             entered or if a negative number is entered; if a negative
             value is entered, the (positive) "count" structures that
             lead up to and include the target structure will be displayed.

Structure data, sizes, and member offsets are shown in the current output
radix unless the -x or -d option is specified.

Please note that in the vast majority of cases, the “struct” command
name may be dropped; if the structure name does not conflict with any crash
or gdb command name, then the “struct_name[.member]” argument will be
recognized as a structure name, and this command automatically executed.
See the NOTE below.

EXAMPLES
Display the vm_area_struct at address c1e44f10:

crash> struct vm_area_struct c1e44f10
struct vm_area_struct {
  vm_mm = 0xc2857750,
  vm_start = 0x8048000, 
  vm_end = 0x80a5000, 
  vm_next = 0xc1e44a10,
  vm_page_prot = {
    pgprot = 0x25      
  },
  vm_flags = 0x1875,
  vm_avl_height = 0x2,   
  vm_avl_left = 0xc30fe200,
  vm_avl_right = 0xc30fed00,
  vm_next_share = 0x0,       
  vm_pprev_share = 0xc1e44a30,
  vm_ops = 0xc0215ca0,
  vm_offset = 0x0,       
  vm_file = 0xc0bfdc70,
  vm_pte = 0   
}

Display the definition and size of a vm_area_struct structure. This first
example below displays just the structure and size. The second example
uses the -o option to also display member offsets. Both examples were
run with the output radix set to 10 (decimal):

crash> struct vm_area_struct
struct vm_area_struct {
    struct mm_struct *vm_mm;
    long unsigned int vm_start;
    long unsigned int vm_end;
    struct vm_area_struct *vm_next;
    pgprot_t vm_page_prot;
    short unsigned int vm_flags;
    short int vm_avl_height;
    struct vm_area_struct *vm_avl_left;
    struct vm_area_struct *vm_avl_right;
    struct vm_area_struct *vm_next_share;
    struct vm_area_struct **vm_pprev_share;
    struct vm_operations_struct *vm_ops;
    long unsigned int vm_offset;
    struct file *vm_file;
    long unsigned int vm_pte;
}
SIZE: 56

crash> struct vm_area_struct -o
struct vm_area_struct {
   [0] struct mm_struct *vm_mm;
   [4] long unsigned int vm_start;
   [8] long unsigned int vm_end;
  [12] struct vm_area_struct *vm_next;
  [16] pgprot_t vm_page_prot;
  [20] short unsigned int vm_flags;
  [22] short int vm_avl_height;
  [24] struct vm_area_struct *vm_avl_left;
  [28] struct vm_area_struct *vm_avl_right;
  [32] struct vm_area_struct *vm_next_share;
  [36] struct vm_area_struct **vm_pprev_share;
  [40] struct vm_operations_struct *vm_ops;
  [44] long unsigned int vm_offset;
  [48] struct file *vm_file;
  [52] long unsigned int vm_pte;
}
SIZE: 56

Display the definition and offset of the pgd member of an mm_struct:

crash> struct mm_struct.pgd
struct mm_struct {
   [80] pgd_t *pgd;
}

Display the pgd member of the mm_struct at address ffff810022e7d080:

crash> struct mm_struct.pgd ffff810022e7d080
  pgd = 0xffff81000e3ac000

Display the pgd_t pointed to by the mm_struct.pgd pointer above, forcing
the output to be expressed in hexadecimal:

crash> mm_struct.pgd ffff810022e7d080 -px
  pgd_t *pgd = 0xffff81000e3ac000
  -> {
       pgd = 0x2c0a6067
     }

Display the thread_info structure pointed to by the thread_info
member of the task_struct at ffff8100181190c0:

crash> task_struct.thread_info ffff8100181190c0 -p
  struct thread_info *thread_info = 0xffff810023c06000
  -> {
       task = 0xffff8100181190c0,
       exec_domain = 0xffffffff802f78e0,
       flags = 128,
       status = 1,
       cpu = 3,
       preempt_count = 0,
       addr_limit = {
         seg = 18446604435732824064
       },
       restart_block = {
         fn = 0xffffffff80095a52 <do_no_restart_syscall>,
         arg0 = 0,
         arg1 = 0,
         arg2 = 0,
         arg3 = 0
       }
     }

Display the flags and virtual members of 4 contigous page structures
in the mem_map page structure array:

crash> page.flags,virtual c101196c 4
  flags = 0x8000,
  virtual = 0xc04b0000

  flags = 0x8000,
  virtual = 0xc04b1000

  flags = 0x8000,
  virtual = 0xc04b2000

  flags = 0x8000,
  virtual = 0xc04b3000

Display the array of tcp_sl_timer structures declared by tcp_slt_array[]:

crash> struct tcp_sl_timer tcp_slt_array 4
struct tcp_sl_timer {
  count = {
    counter = 0x0       
  },
  period = 0x32,      
  last = 0x1419e4,  
  handler = 0xc0164854  <tcp_syn_recv_timer>
}
struct tcp_sl_timer {
  count = {
    counter = 0x2       
  },
  period = 0x753,     
  last = 0x14a6df,  
  handler = 0xc01645b0  <tcp_keepalive>
}
struct tcp_sl_timer {
  count = {
    counter = 0x0       
  },
  period = 0x2ee,     
  last = 0x143134,  
  handler = 0xc016447c  <tcp_twkill>
}
struct tcp_sl_timer {
  count = {
    counter = 0x0       
  },
  period = 0x64,      
  last = 0x143198,  
  handler = 0xc0164404  <tcp_bucketgc>
}

Without using the “struct” command name, display the the “d_child”
list_head member from a dentry structure:

crash> dentry.d_child 0xe813cb4
  d_child = {
    next = 0x3661344,
    prev = 0xdea4bc4
  },

Display the child dentry structure referenced by the “next” pointer above.
Since the “next” address of 0x3661344 above is a pointer to an embedded
list_head structure within the child dentry structure, the -l option
is required:

crash> dentry -l dentry.d_child 0x3661344
struct dentry {
  d_count = {
    counter = 1
  }, 
  d_flags = 0, 
  d_inode = 0xf9aa604, 
  d_parent = 0x11152b1c, 
  d_hash = {
    next = 0x11fb3fc0, 
    prev = 0x11fb3fc0
  }, 
  d_lru = {
    next = 0x366133c, 
    prev = 0x366133c
  }, 
  d_child = {
    next = 0x36613cc, 
    prev = 0xe813cd4
  }, 
  d_subdirs = {
    next = 0x366134c, 
    prev = 0x366134c
  }, 
  d_alias = {
    next = 0xf9aa614, 
    prev = 0xf9aa614
  }, 
  d_mounted = 0, 
  d_name = {
    name = 0x3661384 "boot.log", 
    len = 8, 
    hash = 1935169207
  }, 
  d_time = 1515870810, 
  d_op = 0x0, 
  d_sb = 0x11fc9c00, 
  d_vfs_flags = 0, 
  d_fsdata = 0x0, 
  d_extra_attributes = 0x0, 
  d_iname = "boot.log\000"
}

Display the virtual address of each member of the task_struct at
ffff8100145d2080:

crash> task_struct -o ffff8100145d2080
struct task_struct {
  [ffff8100145d2080] volatile long int state;
  [ffff8100145d2088] struct thread_info *thread_info;
  [ffff8100145d2090] atomic_t usage;
  [ffff8100145d2098] long unsigned int flags;
  [ffff8100145d20a0] int lock_depth;
  [ffff8100145d20a4] int load_weight;
  [ffff8100145d20a8] int prio;
  [ffff8100145d20ac] int static_prio;
  [ffff8100145d20b0] int normal_prio;
  [ffff8100145d20b8] struct list_head run_list;
  [ffff8100145d20c8] struct prio_array *array;
...

Display the embedded sched_entity structure’s on_rq member and
the third pid_link structure in the embedded pids[] array of the
task_struct at ffff88011653e250:

crash> task_struct.se.on_rq,pids[2] ffff88011653e250
  se.on_rq = 1,
  pids[2] =   {
    node = {
      next = 0xffff88011653aff0,
      pprev = 0xffff88011653a860
    },
    pid = 0xffff88010d07ed00
  }

For an example of displaying per-cpu variables, consider the
struct hd_struct.dkstats member, which is a percpu pointer to
a disk_stats structure:

crash> struct hd_struct.dkstats 
struct hd_struct {
  [1232] struct disk_stats *dkstats;
}

Taking an hd_struct at address ffff8802450e2848, display all
of the per-cpu disk_stats structures that it references:

crash> struct hd_struct.dkstats ffff8802450e2848
  dkstats = 0x60fdb48026c8
crash> struct disk_stats 0x60fdb48026c8:a
[0]: ffffe8fefe6026c8
struct disk_stats {
  sectors = {451376, 80468}, 
  ios = {6041, 971}, 
  merges = {386, 390}, 
  ticks = {194877, 56131}, 
  io_ticks = 12371, 
  time_in_queue = 309163
}
[1]: ffffe8fefe8026c8
struct disk_stats {
  sectors = {0, 0}, 
  ios = {0, 0}, 
  merges = {7, 242}, 
  ticks = {0, 0}, 
  io_ticks = 23, 
  time_in_queue = 581
}
[2]: ffffe8fefea026c8
struct disk_stats {
  sectors = {0, 0}, 
  ios = {0, 0}, 
  merges = {4, 112}, 
  ticks = {0, 0}, 
  io_ticks = 11, 
  time_in_queue = 305
}
[3]: ffffe8fefec026c8
struct disk_stats {
  sectors = {0, 0}, 
  ios = {0, 0}, 
  merges = {5, 54}, 
  ticks = {0, 0}, 
  io_ticks = 17, 
  time_in_queue = 41
}

NOTE
If the structure name does not conflict with any crash command name, the
“struct” command may be dropped. Accordingly, the examples above could
also have been accomplished like so:

crash> vm_area_struct c1e44f10
crash> vm_area_struct
crash> vm_area_struct -o
crash> mm_struct.pgd ffff810022e7d080
crash> mm_struct.pgd
crash> tcp_sl_timer tcp_slt_array 4

Lastly, the short-cut “*” pointer-to command may also be used to negate
the need to enter the “struct” command name (enter “help *” for details).
//////////////////////////////////////////////////////////////////////////////////

NAME

waitq - list tasks queued on a wait queue

SYNOPSIS
waitq [ symbol ] | [ struct.member struct_addr ] | [ address ]

DESCRIPTION
This command walks the wait queue list displaying the tasks which
are blocked on the specified wait queue. The command differentiates
between the old- and new-style wait queue structures used by the kernel.
It can be invoked with the following argument types:

                 symbol  a global symbol of a wait queue.

struct.member struct_addr a structure name and wait queue member combination
followed by the structure’s hexadecimal address.
address a hexadecimal wait queue pointer.

EXAMPLES

Find out if any tasks are blocked on the “buffer_wait” wait queue:

crash> waitq buffer_wait
wait queue "buffer_wait" (c02927f0) is empty

See who is blocked on the “wait_chldexit” queue of task c5496000:

crash> waitq task_struct.wait_chldexit c5496000
PID: 30879  TASK: c5496000  CPU: 0   COMMAND: "bash"

Display the task list waiting on a known task queue:

crash> waitq c3534098
PID: 13691  TASK: c3534000  CPU: 1   COMMAND: "bash"

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

dis - disassemble

SYNOPSIS
dis [-rfludxs][-b [num]] [address | symbol | (expression)] [count]

DESCRIPTION
This command disassembles source code instructions starting (or ending) at
a text address that may be expressed by value, symbol or expression:

        -r  (reverse) displays all instructions from the start of the 
            routine up to and including the designated address.
        -f  (forward) displays all instructions from the given address 
            to the end of the routine.
        -l  displays source code line number data in addition to the 
            disassembly output.
        -u  address is a user virtual address in the current context;
            otherwise the address is assumed to be a kernel virtual address.
            If this option is used, then -r and -l are ignored.
        -x  override default output format with hexadecimal format.
        -d  override default output format with decimal format.
        -s  displays the filename and line number of the source code that
            is associated with the specified text location, followed by a
            source code listing if it is available on the host machine.
            The line associated with the text location will be marked with
            an asterisk; depending upon gdb's internal "listsize" variable,
            several lines will precede the marked location. If a "count"
            argument is entered, it specifies the number of source code
            lines to be displayed after the marked location; otherwise
            the remaining source code of the containing function will be
            displayed.
  -b [num]  modify the pre-calculated number of encoded bytes to skip after
            a kernel BUG ("ud2a") instruction; with no argument, displays
            the current number of bytes being skipped. (x86 and x86_64 only)
   address  starting hexadecimal text address.
    symbol  symbol of starting text address.  On ppc64, the symbol
            preceded by '.' is used.

(expression) expression evaluating to a starting text address.
count the number of instructions to be disassembled (default is 1).
If no count argument is entered, and the starting address
is entered as a text symbol, then the whole routine will be
disassembled. The count argument is supported when used with
the -r and -f options.

EXAMPLES
Disassemble the sys_signal() routine without, and then with, line numbers:

crash> dis sys_signal
0xc0112c88 <sys_signal>:        push   %ebp
0xc0112c89 <sys_signal+1>:      mov    %esp,%ebp
0xc0112c8b <sys_signal+3>:      sub    $0x28,%esp
0xc0112c8e <sys_signal+6>:      mov    0xc(%ebp),%eax
0xc0112c91 <sys_signal+9>:      mov    %eax,0xffffffec(%ebp)
0xc0112c94 <sys_signal+12>:     movl   $0xc0000000,0xfffffff0(%ebp)
0xc0112c9b <sys_signal+19>:     lea    0xffffffd8(%ebp),%eax
0xc0112c9e <sys_signal+22>:     push   %eax
0xc0112c9f <sys_signal+23>:     lea    0xffffffec(%ebp),%eax
0xc0112ca2 <sys_signal+26>:     push   %eax
0xc0112ca3 <sys_signal+27>:     pushl  0x8(%ebp)
0xc0112ca6 <sys_signal+30>:     call   0xc01124b8 <do_sigaction>
0xc0112cab <sys_signal+35>:     test   %eax,%eax
0xc0112cad <sys_signal+37>:     jne    0xc0112cb2 <sys_signal+42>
0xc0112caf <sys_signal+39>:     mov    0xffffffd8(%ebp),%eax
0xc0112cb2 <sys_signal+42>:     leave
0xc0112cb3 <sys_signal+43>:     ret

crash> dis -l sys_signal
/usr/src/linux-2.2.5/kernel/signal.c: 1074
0xc0112c88 <sys_signal>:        push   %ebp
0xc0112c89 <sys_signal+1>:      mov    %esp,%ebp
0xc0112c8b <sys_signal+3>:      sub    $0x28,%esp
0xc0112c8e <sys_signal+6>:      mov    0xc(%ebp),%eax
/usr/src/linux-2.2.5/kernel/signal.c: 1078
0xc0112c91 <sys_signal+9>:      mov    %eax,0xffffffec(%ebp)
/usr/src/linux-2.2.5/kernel/signal.c: 1079
0xc0112c94 <sys_signal+12>:     movl   $0xc0000000,0xfffffff0(%ebp)
/usr/src/linux-2.2.5/kernel/signal.c: 1081
0xc0112c9b <sys_signal+19>:     lea    0xffffffd8(%ebp),%eax
0xc0112c9e <sys_signal+22>:     push   %eax
0xc0112c9f <sys_signal+23>:     lea    0xffffffec(%ebp),%eax
0xc0112ca2 <sys_signal+26>:     push   %eax
0xc0112ca3 <sys_signal+27>:     pushl  0x8(%ebp)
0xc0112ca6 <sys_signal+30>:     call   0xc01124b8 <do_sigaction>
/usr/src/linux-2.2.5/kernel/signal.c: 1083
0xc0112cab <sys_signal+35>:     test   %eax,%eax
0xc0112cad <sys_signal+37>:     jne    0xc0112cb2 <sys_signal+42>
0xc0112caf <sys_signal+39>:     mov    0xffffffd8(%ebp),%eax
/usr/src/linux-2.2.5/kernel/signal.c: 1084
0xc0112cb2 <sys_signal+42>:     leave
0xc0112cb3 <sys_signal+43>:     ret

Given a return address expression of “do_no_page+65”, find out the
function that do_no_page() calls by using the reverse flag:

crash> dis -r (do_no_page+65)
0xc011ea68 <do_no_page>:        push   %ebp
0xc011ea69 <do_no_page+1>:      mov    %esp,%ebp
0xc011ea6b <do_no_page+3>:      push   %edi
0xc011ea6c <do_no_page+4>:      push   %esi
0xc011ea6d <do_no_page+5>:      push   %ebx
0xc011ea6e <do_no_page+6>:      mov    0xc(%ebp),%ebx
0xc011ea71 <do_no_page+9>:      mov    0x10(%ebp),%edx
0xc011ea74 <do_no_page+12>:     mov    0x14(%ebp),%edi
0xc011ea77 <do_no_page+15>:     mov    0x28(%ebx),%eax
0xc011ea7a <do_no_page+18>:     test   %eax,%eax
0xc011ea7c <do_no_page+20>:     je     0xc011ea85 <do_no_page+29>
0xc011ea7e <do_no_page+22>:     mov    0x18(%eax),%ecx
0xc011ea81 <do_no_page+25>:     test   %ecx,%ecx
0xc011ea83 <do_no_page+27>:     jne    0xc011eab0 <do_no_page+72>
0xc011ea85 <do_no_page+29>:     mov    $0xffffe000,%eax
0xc011ea8a <do_no_page+34>:     and    %esp,%eax
0xc011ea8c <do_no_page+36>:     decl   0x30(%eax)
0xc011ea8f <do_no_page+39>:     jns    0xc011ea9a <do_no_page+50>
0xc011ea91 <do_no_page+41>:     lock btrl $0x0,0xc022fb60
0xc011ea9a <do_no_page+50>:     push   %edi
0xc011ea9b <do_no_page+51>:     mov    0x18(%ebp),%esi
0xc011ea9e <do_no_page+54>:     push   %esi
0xc011ea9f <do_no_page+55>:     push   %ebx
0xc011eaa0 <do_no_page+56>:     mov    0x8(%ebp),%esi
0xc011eaa3 <do_no_page+59>:     push   %esi
0xc011eaa4 <do_no_page+60>:     call   0xc011e9e4 <do_anonymous_page>
0xc011eaa9 <do_no_page+65>:     jmp    0xc011eb47 <do_no_page+223>

Disassemble 10 instructions starting at user virtual address 0x81ec624:

crash> dis -u 81ec624 10
0x81ec624:      push   %ebp
0x81ec625:      mov    %esp,%ebp
0x81ec627:      sub    $0x18,%esp
0x81ec62a:      movl   $0x1,0x8(%ebp)
0x81ec631:      mov    0x82f9040,%eax
0x81ec636:      mov    0x10(%eax),%edx
0x81ec639:      and    $0x100,%edx
0x81ec63f:      mov    0x14(%eax),%ecx
0x81ec642:      and    $0x0,%ecx
0x81ec645:      mov    %ecx,%eax

Override the current decimal output radix format:

crash> dis sys_read 10 -x
0xffffffff8001178f <sys_read>:  push   %r13
0xffffffff80011791 <sys_read+0x2>:    mov    %rsi,%r13
0xffffffff80011794 <sys_read+0x5>:    push   %r12
0xffffffff80011796 <sys_read+0x7>:    mov    $0xfffffffffffffff7,%r12
0xffffffff8001179d <sys_read+0xe>:    push   %rbp
0xffffffff8001179e <sys_read+0xf>:    mov    %rdx,%rbp
0xffffffff800117a1 <sys_read+0x12>:   push   %rbx
0xffffffff800117a2 <sys_read+0x13>:   sub    $0x18,%rsp
0xffffffff800117a6 <sys_read+0x17>:   lea    0x14(%rsp),%rsi
0xffffffff800117ab <sys_read+0x1c>:   callq  0xffffffff8000b5b4 <fget_light>

Disassemble from vfs_read+320 until the end of the function:

crash> dis -f vfs_read+320
0xffffffff8119d4e0 <vfs_read+320>:  cmpq   $0x0,0x20(%rax)
0xffffffff8119d4e5 <vfs_read+325>:  jne    0xffffffff8119d3e8 <vfs_read+72>
0xffffffff8119d4eb <vfs_read+331>:  mov    $0xffffffffffffffea,%r12
0xffffffff8119d4f2 <vfs_read+338>:  jmp    0xffffffff8119d4c3 <vfs_read+291>
0xffffffff8119d4f4 <vfs_read+340>:  nopl   0x0(%rax)
0xffffffff8119d4f8 <vfs_read+344>:  callq  0xffffffff8119cc40 <do_sync_read>
0xffffffff8119d4fd <vfs_read+349>:  mov    %rax,%r12
0xffffffff8119d500 <vfs_read+352>:  jmpq   0xffffffff8119d44c <vfs_read+172>
0xffffffff8119d505 <vfs_read+357>:  nopl   (%rax)
0xffffffff8119d508 <vfs_read+360>:  mov    $0xfffffffffffffff7,%r12
0xffffffff8119d50f <vfs_read+367>:  jmp    0xffffffff8119d4c3 <vfs_read+291>
0xffffffff8119d511 <vfs_read+369>:  mov    $0xfffffffffffffff2,%r12
0xffffffff8119d518 <vfs_read+376>:  jmp    0xffffffff8119d4c3 <vfs_read+291>
0xffffffff8119d51a <vfs_read+378>:  nopw   0x0(%rax,%rax,1)

Display the source code listing of the mmput() function:

crash> dis -s mmput
FILE: kernel/fork.c
LINE: 617

  612   
  613   /*
  614    * Decrement the use count and release all resources for an mm.
  615    */
  616   void mmput(struct mm_struct *mm)
* 617   {
  618           might_sleep();
  619   
  620           if (atomic_dec_and_test(&mm->mm_users)) {
  621                   uprobe_clear_state(mm);
  622                   exit_aio(mm);
  623                   ksm_exit(mm);
  624                   khugepaged_exit(mm); /* must run before exit_mmap */
  625                   exit_mmap(mm);
  626                   set_mm_exe_file(mm, NULL);
  627                   if (!list_empty(&mm->mmlist)) {
  628                           spin_lock(&mmlist_lock);
  629                           list_del(&mm->mmlist);
  630                           spin_unlock(&mmlist_lock);
  631                   }
  632                   if (mm->binfmt)
  633                           module_put(mm->binfmt->module);
  634                   mmdrop(mm);
  635           }
  636   }

The disassembly of dentry_kill() shows an indirect call to a function
whose address is contained within a register. Display the source code
associated with the indirect function call:

crash> dis dentry_kill
...
0xffffffff811dcfb4 <dentry_kill+324>:	callq  *%rax
...
crash> dis -s 0xffffffff811dcfb4
FILE: fs/dcache.c
LINE: 276

  271                   spin_unlock(&dentry->d_lock);
  272                   spin_unlock(&inode->i_lock);
  273                   if (!inode->i_nlink)
  274                           fsnotify_inoderemove(inode);
  275                   if (dentry->d_op && dentry->d_op->d_iput)
* 276                           dentry->d_op->d_iput(dentry, inode);
  277                   else
  278                           iput(inode);
  279           } else {
  280                   spin_unlock(&dentry->d_lock);
  281           }
  282   }

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

irq - IRQ data

SYNOPSIS
irq [[[index …] | -u ] | -d | -b | -a | -s [-c cpu]]

DESCRIPTION
This command collaborates the data in an irq_desc_t, along with its
associated hw_interrupt_type and irqaction structure data, into a
consolidated per-IRQ display. For kernel versions 2.6.37 and later
the display consists of the irq_desc/irq_data address, its irqaction
address(es), and the irqaction name strings. Alternatively, the
intel interrupt descriptor table, bottom half data, cpu affinity for
in-use irqs, or kernel irq stats may be displayed. If no index value
argument(s) nor any options are entered, the IRQ data for all IRQs will
be displayed.

index   a valid IRQ index.
   -u   dump data for in-use IRQs only.
   -d   dump the intel interrupt descriptor table.
   -b   dump bottom half data.
   -a   dump cpu affinity for in-use IRQs.
   -s   dump the kernel irq stats; if no cpu specified with -c, the
        irq stats of all cpus will be displayed.
   -c cpu   only usable with the -s option, dump the irq stats of the 
        specified cpu[s]; cpu can be specified as "1,3,5", "1-3",
        "1,3,5-7,10", "all", or "a" (shortcut for "all").

EXAMPLES
Display the relevant data for IRQ 18 from a pre-2.6.37 kernel:

crash> irq 18
    IRQ: 18
 STATUS: 0 
HANDLER: c02301e0  <ioapic_level_irq_type>
         typename: c01f9e0c  "IO-APIC-level"
          startup: c0110234  <unmask_IO_APIC_irq>
         shutdown: c01101cc  <mask_IO_APIC_irq>
           handle: c0110518  <do_level_ioapic_IRQ>
           enable: c0110234  <unmask_IO_APIC_irq>
          disable: c01101cc  <mask_IO_APIC_irq>
 ACTION: c009c6b0
          handler: c01ce818  <do_aic7xxx_isr>
            flags: 4000000  (SA_SHIRQ)
             mask: 0
             name: c0217780  "aic7xxx"
           dev_id: c0090078
             next: c009c770
 ACTION: c009c770
          handler: c01ce818  <do_aic7xxx_isr>
            flags: 4000000  (SA_SHIRQ)
             mask: 0
             name: c0217780  "aic7xxx"
           dev_id: c0091078
             next: 0
  DEPTH: 0

Display the relevant data for IRQ 21 from a 2.6.37 kernel:

crash> irq 21
 IRQ   IRQ_DESC/_DATA      IRQACTION      NAME
 21   ffff88003787f780  ffff8800379a8b40  "ehci_hcd:usb2"
                        ffff8800379cbac0  "uhci_hcd:usb5"
                        ffff8800379cb140  "uhci_hcd:usb7"

Display the intel interrupt descriptor table entries:

crash> irq -d
  [0] divide_error
  [1] debug
  [2] nmi
  [3] int3
  [4] overflow
  [5] bounds
  [6] invalid_op
  [7] device_not_available
  [8] double_fault
  [9] coprocessor_segment_overrun
 [10] invalid_TSS
 [11] segment_not_present
 [12] stack_segment
 [13] general_protection
 [14] page_fault
 [15] spurious_interrupt_bug
 [16] coprocessor_error
 [17] alignment_check
 [18] ignore_int
 [19] ignore_int
 [20] ignore_int
 [21] ignore_int
...

[250] IRQ0xda_interrupt
[251] IRQ0xdb_interrupt
[252] IRQ0xdc_interrupt
[253] IRQ0xdd_interrupt
[254] IRQ0xde_interrupt
[255] spurious_interrupt

Display the bottom half data:

crash> irq -b
SOFTIRQ_VEC      ACTION     
    [0]     ffffffff81068f60  <tasklet_hi_action> 
    [1]     ffffffff81071b80  <run_timer_softirq> 
    [2]     ffffffff813e6f30  <net_tx_action> 
    [3]     ffffffff813ee370  <net_rx_action> 
    [4]     ffffffff81211a60  <blk_done_softirq> 
    [5]     ffffffff812122f0  <blk_iopoll_softirq> 
    [6]     ffffffff81069090  <tasklet_action> 
    [7]     ffffffff81058830  <run_rebalance_domains> 
    [8]     ffffffff81087f00  <run_hrtimer_softirq> 
    [9]     ffffffff810ca7a0  <rcu_process_callbacks> 

Display the cpu affinity for in-use IRQs:

crash> irq -a
IRQ NAME                 AFFINITY
  0 timer                0-23
  1 i8042                0-23
  8 rtc0                 0-23
  9 acpi                 0-23
 16 ehci_hcd:usb2,uhci_hcd:usb3,uhci_hcd:usb6 0,6,18
 17 uhci_hcd:usb4,uhci_hcd:usb7 0-23
 18 ehci_hcd:usb1,uhci_hcd:usb5,uhci_hcd:usb8,ioc0 0,11,23
 24 dmar0                0
 35 pciehp               0-23
 36 pciehp               0-23
 37 pciehp               0-23
 38 pciehp               0-23
 39 megasas              0-5,12-17
 40 lpfc:sp              0-5,12-17
 41 lpfc:fp              0,6-11,18-23
 42 lpfc:sp              0,6-11,18-23
 43 lpfc:fp              0,6-11,18-23
...

 80 ioat-msix            0-23
 81 ioat-msix            0-23
 82 ioat-msix            0-23
 83 ioat-msix            0-23
 84 ioat-msix            0-23
 85 ioat-msix            0-23
 86 ioat-msix            0-23
 87 ioat-msix            0-23
 88 eth4                 0,17

Display the kernel irq stats:

crash>irq -c 0,2 -s
           CPU0       CPU2 
  0: 2068161471          0 IR-IO-APIC-edge     timer
  1:          9          0 IR-IO-APIC-edge     i8042
  8:          1          0 IR-IO-APIC-edge     rtc0
  9:          0          0 IR-IO-APIC-fasteoi  acpi
 16:         36          0 IR-IO-APIC-fasteoi  ehci_hcd:usb2
...

 85:          3          0 IR-PCI-MSI-edge     ioat-msix
 86:          3          0 IR-PCI-MSI-edge     ioat-msix
 87:          3          0 IR-PCI-MSI-edge     ioat-msix
 88:         24        295 IR-PCI-MSI-edge     eth4

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

pte - translate a page table entry

SYNOPSIS
pte contents …

DESCRIPTION
This command translates the hexadecimal contents of a PTE into its physical
page address and page bit settings. If the PTE references a swap location,
the swap device and offset are displayed.

EXAMPLES

crash> pte d8e067
 PTE    PHYSICAL  FLAGS
d8e067   d8e000   (PRESENT|RW|USER|ACCESSED|DIRTY)

crash> pte 13f600
 PTE      SWAP     OFFSET
13f600  /dev/hda2   5104

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

swap - swap device information

SYNOPSIS
swap

DESCRIPTION
This command displays information for each configured swap device.

EXAMPLE

  crash> swap
  SWAP_INFO_STRUCT    TYPE       SIZE       USED    PCT  PRI  FILENAME
  ffff880153d45f40  PARTITION  7192568k   1200580k  16%   -1  /dev/dm-1

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

whatis - search symbol table for data or type information

SYNOPSIS
whatis [[-o] [struct | union | typedef | symbol]] |
[[-r [size|range]] [-m member]]

DESCRIPTION
This command displays the definition of structures, unions, typedefs or
text/data symbols:

struct  a structure name. The output is the same as if the "struct"
        command was used.
 union  a union name. The output is the same as if the "union" command
        was used.
    -o  display the offsets of structure/union members.

typedef a typedef name. If the typedef translates to a structure or union
the output is the same as if the “struct” or “union” command
was used. If the typedef is a primitive datatype, the one-line
declaration is displayed.
symbol a kernel symbol.

Alternatively, a search can be made for data structures of a given size or
size range, that contain a member of a given type, or contain a pointer to
given type. The -r and -m options may be used alone or in conjunction with
one another:

   -r size  search for structures of this exact size.
   -r range  search for structures of a range of sizes, expressed as "low-high".
   -m member  search for structures that contain a member of this data type, or
        that contain a pointer to this data type; if a structure contains 
        another structure, the members of the embedded structure will also
        be subject to the search. The member argument may also be expressed
        as a substring of a member's data type.

EXAMPLES
Display the definition of a linux_binfmt structure:

crash> whatis linux_binfmt
struct linux_binfmt {
    struct list_head lh;
    struct module *module;
    int (*load_binary)(struct linux_binprm *);
    int (*load_shlib)(struct file *);
    int (*core_dump)(struct coredump_params *);
    unsigned long min_coredump;
}
SIZE: 56

Display the same structure with member offsets:

crash> whatis -o linux_binfmt
struct linux_binfmt {
   [0] struct list_head lh;
  [16] struct module *module;
  [24] int (*load_binary)(struct linux_binprm *);
  [32] int (*load_shlib)(struct file *);
  [40] int (*core_dump)(struct coredump_params *);
  [48] unsigned long min_coredump;
}
SIZE: 56

Since a kmem_bufctl_t is typedef’d to be a kmem_bufctl_s structure, the
output of the following two commands is identical:

crash> whatis kmem_bufctl_s
struct kmem_bufctl_s {
  union {
    struct kmem_bufctl_s  *buf_nextp;
    kmem_slab_t *buf_slabp;
    void *buf_objp;
  } u;
};

crash> whatis kmem_bufctl_t
struct kmem_bufctl_s {
  union {
    struct kmem_bufctl_s *buf_nextp;
    kmem_slab_t *buf_slabp;
    void *buf_objp;
  } u;
};
SIZE: 4  (0x4)

Display the type data of sys_read() and jiffies text and data symbols:

crash> whatis sys_read
ssize_t sys_read(unsigned int, char *, size_t);

crash> whatis jiffies
long unsigned int jiffies;

Display definition of a kdev_t typedef:

crash> whatis kdev_t
typedef short unsigned int kdev_t;
SIZE: 2  (0x2)

Display all structures which have a size of 192 bytes:

crash> whatis -r 192
SIZE  TYPE
 192  _intel_private
 192  blkcg_gq
 192  clock_event_device
 192  cper_sec_proc_generic
 192  dentry
 192  dst_ops
 192  ehci_itd
 192  ethtool_rxnfc
 192  fb_ops
 192  file_lock
 192  inode_operations
 192  input_device_id
 192  ip_vs_stats
 192  numa_group
 192  parallel_data
 192  pcie_port_service_driver
 192  pebs_record_hsw
 192  pnp_driver
 192  regmap_config
 192  sched_entity
 192  tcp_timewait_sock
 192  timerfd_ctx
 192  tpm_vendor_specific
 192  urb

Display all structures that contain members that point to
an mm_struct:

crash> whatis -m mm_struct
SIZE  TYPE
  16  tlb_state
  24  flush_tlb_info
  24  ftrace_raw_xen_mmu_pgd
  24  futex_key
  24  map_info
  32  ftrace_raw_xen_mmu_alloc_ptpage
  32  ftrace_raw_xen_mmu_pte_clear
  40  ftrace_raw_xen_mmu_flush_tlb_others
  40  ftrace_raw_xen_mmu_ptep_modify_prot
  40  ftrace_raw_xen_mmu_set_pte_at
  40  mm_slot
  64  mm_walk
  64  rmap_item
 104  userfaultfd_ctx
 128  mmu_gather
 216  vm_area_struct
 256  linux_binprm
2616  rq
2936  task_struct

Display all structures sized from 256 to 512 bytes that
contain members that point to a task_struct:

crash> whatis -r 256-512 -m task_struct
SIZE  TYPE
 256  file
 256  od_cpu_dbs_info_s
 264  srcu_notifier_head
 272  protection_domain
 288  clk_notifier
 288  fsnotify_group
 296  quota_info
 312  tty_port
 320  workqueue_struct
 344  trace_array
 344  uart_state
 352  cpufreq_policy
 352  elf_thread_core_info
 376  perf_event_context
 384  rcu_data
 400  cgroup
 408  subsys_private
 424  hvc_struct
 496  psmouse

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

eval - evaluate

SYNOPSIS
eval [-b][-l] (expression) | value

DESCRIPTION
This command evaluates an expression or numeric value, and displays its
result in hexadecimal, decimal, octal and binary. If the resultant value
is an integral number of gigabytes, megabytes, or kilobytes, a short-hand
translation of the number will also be shown next to the hexadecimal
value. If the most significant bit is set, the decimal display will show
both unsigned and signed (negative) values. Expressions must of the format
(x operator y), where “x” and “y” may be either numeric values or
symbols. The list of operators are:

                 +  -  &  |  ^  *  %  /  <<  >>

Enclosing the expression within parentheses is optional except when the
“|”, “<<” or “>>” operators are used. The single “value” argument may
be a number or symbol. Number arguments must be hexadecimal or decimal.
A leading “0x” identifies a number as hexadecimal, but is not required
when obvious. Numbers may be followed by the letters “k” or “K”, “m”
or “M”, and “g” or “G”, which multiplies the value by a factor of 1024,
1 megabyte or 1 gigabyte, respectively. Numeric arguments may be preceded
by the one’s complement operator ~.

-b  Indicate which bit positions in the resultant value are set.
-l  Numeric arguments are presumed to be 64-bit values, and the result
    will be expressed as a 64-bit value. (ignored on 64-bit processors)
    However, if either operand or the resultant value are 64-bit values,
    then the result will be also be expressed as a 64-bit value.

The -b and -l options must precede the expression or value arguments.

EXAMPLES

crash> eval 128m
   hexadecimal: 8000000  (128MB)
       decimal: 134217728  
         octal: 1000000000
        binary: 00001000000000000000000000000000

crash> eval 128 * 1m
hexadecimal: 8000000  (128MB)
   decimal: 134217728  
     octal: 1000000000
    binary: 00001000000000000000000000000000

crash> eval (1 << 27)
hexadecimal: 8000000  (128MB)
   decimal: 134217728  
     octal: 1000000000
    binary: 00001000000000000000000000000000

crash> eval (1 << 32)
hexadecimal: 100000000  (4GB)
   decimal: 4294967296
     octal: 40000000000
    binary: 0000000000000000000000000000000100000000000000000000000000000000

crash> eval -b 41dc065
hexadecimal: 41dc065
   decimal: 69058661  
     octal: 407340145
    binary: 00000100000111011100000001100101
  bits set: 26 20 19 18 16 15 14 6 5 2 0 

crash> eval -lb 64g
hexadecimal: 1000000000  (64GB)
   decimal: 68719476736
     octal: 1000000000000
    binary: 0000000000000000000000000001000000000000000000000000000000000000
  bits set: 36

///////////////////////////////////////////////////////////////////////////////////////////////////

NAME

kmem - kernel memory

SYNOPSIS
kmem [-f|-F|-c|-C|-i|-v|-V|-n|-z|-o|-h] [-p | -m member[,member]]
[[-s|-S|-r] [slab] [-I slab[,slab]]] [-g [flags]] [[-P] address]]

DESCRIPTION
This command displays information about the use of kernel memory.

    -f  displays the contents of the system free memory headers.
        also verifies that the page count equals nr_free_pages.
    -F  same as -f, but also dumps all pages linked to that header.
    -c  walks through the page_hash_table and verifies page_cache_size.
    -C  same as -c, but also dumps all pages in the page_hash_table.
    -i  displays general memory usage information
    -v  displays the mapped virtual memory regions allocated by vmalloc().
    -V  displays the kernel vm_stat table if it exists, or in more recent
        kernels, the vm_zone_stat, vm_node_stat and vm_numa_stat tables,
        the cumulative page_states counter values if they exist, and/or 
        the cumulative, vm_event_states counter values if they exist.
    -n  display memory node, memory section, and memory block data
        and state; the state of each memory section state is encoded
        as "P", "M", "O" and/or "E", meaning SECTION_MARKED_PRESENT,
        SECTION_HAS_MEM_MAP, SECTION_IS_ONLINE and SECTION_IS_EARLY.
    -z  displays per-zone memory statistics.
    -o  displays each cpu's offset value that is added to per-cpu symbol
        values to translate them into kernel virtual addresses.
    -h  display the address of hugepage hstate array entries, along with
        their hugepage size, total and free counts, and name.
    -p  displays basic information about each page structure in the system
        mem_map[] array, made up of the page struct address, its associated
        physical address, the page.mapping, page.index, page._count and
        page.flags fields.
    -m member  similar to -p, but displays page structure contents specified by
        a comma-separated list of one or more struct page members.  The
        "flags" member will always be expressed in hexadecimal format, and
        the "_count" and "_mapcount" members will always be expressed
        in decimal format.  Otherwise, all other members will be displayed
        in hexadecimal format unless the output radix is 10 and the member
        is a signed/unsigned integer.  Members that are data structures may
        be specified either by the data structure's member name, or expanded
        to specify a member of the data structure.  For example, "-m lru"
        refers to a list_head data structure, and both the list_head.next
        and list_head.prev pointer values will be displayed, whereas if
        "-m lru.next" is specified, just the list_head.next value will
        be displayed.
    -s  displays basic kmalloc() slab data.
    -S  displays all kmalloc() slab data, including all slab objects,
        and whether each object is in use or is free.  If CONFIG_SLUB,
        slab data for each per-cpu slab is displayed, along with the
        address of each kmem_cache_node, its count of full and partial
        slabs, and a list of all tracked slabs.
    -r  displays the accumulated basic kmalloc() slab data of each
        root slab cache and its children.  The kernel must contain the
        "slab_root_caches" list_head. (currently only available if
        CONFIG_SLUB)

slab when used with -s, -S or -r, limits the command to only the slab
cache of name “slab”. If the slab argument is “list”, then
all slab cache names and addresses are listed.
-I slab when used with -s, -S or -r, one or more slab cache names in a
comma-separated list may be specified as slab caches to ignore.
-

发布了6 篇原创文章 · 获赞 0 · 访问量 212
展开阅读全文

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 大白 设计师: CSDN官方博客

分享到微信朋友圈

×

扫一扫,手机浏览