Linux Kernel Crash Dump note

how to use crash utility to analyze vmcore files.

内核调试方法

使用crash工具查看定位linux内核崩溃原因：

[root@mi_cct vmcoredump]# crash /usr/lib/debug/usr/lib/modules/3.10.0-1062.el7.x86_64/vmlinux vmcore

确定基本信息后，定位基本奔溃原因；

表示信息解释如下：

KERNEL: specifies the kernel running at the time of the crash.
DUMPFILE: is the name of the dumped memory core.
CPUS: is the number of CPUs on your machine.
DATE: specifies the time of the crash.
TASKS: indicates the number of tasks in the memory at the time of the crash. Task is a set of program instructions loaded into memory.
NODENAME: is the name of the crashed host.
RELEASE: and VERSION: specify the kernel release and version.
MACHINE: specifies the architecture of the CPU.
MEMORY: is the size of the physical memory on the crashed machine. And now come the interesting bits:
PANIC: specifies what kind of crash occurred on the machine. There are several types that you can see.

Panic是一种表示系统遇到致命且无法恢复的错误状态。一般导致此状态的原因是程序尝试访问权限以外的地址空间，强制加载或者卸载内核模块，或者硬件问题。

MEMORY:128M
PANIC:"Oops:0002" (check log for details)
PID:1696
COMMAND:"insmod"

Oops:0002；The four digits are a decimal code of the Kernel Page Error.

Kernel page error code

	value
Bit	0	1
0	No page found	Invalid access
1	Read or Excute	Write
2	Kernel mode	user mode
3	Not instruction fetch	Instruction fetch

If the first bit is clear (0), the exception was caused by an access to a page that is not present; if the bit is set (1), this means invalid access right.
If the second bit is clear (0), the exception was caused by read or execute access; if set (1), the exception was caused by a write access.
If the third bit is clear (0), the exception was caused while the processor was in Kernel mode; otherwise, it occurred in User mode.
The fourth bit tells us whether the fault was an Instruction Fetch. This is only valid for 64-bit architecture. Since our machine is 64-bit, the bit has meaning here.

0002 -> 二进制0010 -> 对应表从右到左：Not instruction fetch | Kernel Mode | Write|No page found      meaning no page was found during a write operation in kernel mode.

SysRq (System Request) refers to Magic Keys, which allow you to send instructions directly to the kernel. They can be invoked using a keyboard sequence or by echoing letter commands to /proc/sysrq-trigger, provided the functionality is enabled.

PID: is the process ID of the ... process that caused the crash.
COMMAND: is the name of the process, in this case McsfContainerBa.

TASK: is the address in memory for the offending process. There’s a difference in the memory addressing for 32-bit and 64-bit architectures.
CPU: is the number of the CPU (relevant if more than one) where the offending process was running at the time of the crash. CPU refers to CPU cores and not just physical CPUs. If you’re running your Linux with hyperthreading enabled, then you will also be counting separate threads as CPUs. This is important to remember, because recurring crashes on just one specific CPU might indicate a CPU problem.
If you’re running your processes with affinity set to certain CPUs (taskset), then you might have more difficulty pinpointing CPU-related problems when analyzing the crash reports.
You can examine the number of your CPUs by running cat /proc/cpuinfo.
STATE: indicates the process state at the time of the crash. TASK_RUNNING refers to runnable processes, i.e. processes that can continue their execution. 

调用bt命令 查看堆栈信息

exception RIP:RIP is the instruction pointer.It points to a memory address, indicating the progress of program execution in memory.

可以直接定位到导致崩溃的内核函数以及对应的偏移量；

在堆栈信息之间的段表示寄存器信息，大多数情况下，这些信息用处不大，但是CS(Code Segment)是需要关注的一个寄存器。表示的是一个二进制数值。

CS:0010

代码段（CS）寄存器是指向设置程序指令的段的寄存器。后两位表示的是对CPU当前访问权限等级(CPL)，可以表示0-3四种等级，Level0表示Kernel Level，Level3表示User mode。通常可以简化判断为，最右位是偶数，表示内核态，奇数则表示处于用户态。因此，0010表示运行在内核态过程中奔溃的。

举例来说，如果内核奔溃时CPL值为3(0011、0033，User mode)，可以推断是硬件错误，因为内核不应该因为用用户模式的异常而奔溃的，当然，也有可能是异常的系统调用引起的。

foreach bt 命令用于查看所有task的堆栈信息；

log 命令可以用于查看按照时间顺序排列的内核日志信息；

内核log bugger (logbuf命令）可能在崩溃之前包含有用的线索，可能帮助我们更容易地找出问题所在，并理解我们的系统为何会奔溃下。如果间断性硬件问题或纯粹的软件错误，log命令可能不是很有用的工具，但这绝对值得一试。

ps - display process status information，显示的list中被[]内的是内核线程；

在加载了内核dump文件并使用bt大致定位到问题原因后，想要进一步定位出问题的具体代码就需要做反汇编工程了。

linux下使用 objdump工具将二进制工程文件转换成汇编代码;Objdump user

objdump -d -S null-pointer.ko > /tmp/whatever

通过汇编代码的指令操作，结合dump中看到的寄存器值等堆栈信息进行分析。

In a nutshell, you will execute whatis command against the function listed in exception RIP to see what kind of object it is and what arguments it takes. Then, you will run bt -f command to show all stack data in a frame and focus on the last thing pushed on the stack. After that, you will use stack command to dump the complete contents of the data structure at the given address and work your way through the structure chain, trying to pinpoint the failing/buggy bit of code. MORE CASE STUDY

kerneloops homepage

how to google for information?

The simplest way to search for data is to paste the exception RIP into the search box and look for mailing list threads and forum posts discussing same or similar items.

linux系统奔溃之vmcore：kdump 的亲密战友 crash

解析vmcore文件