OOPS以及【no_context】内核模式处理异常

目录

相关网址

举个例子

OOPS官方文档

---

相关网址

https://www.opensourceforu.com/2011/01/understanding-a-kernel-oops/

举个例子

no_context

#一些异常处理的尝试，不ok就会oops

-show_fault_oops

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305110] BUG: unable to handle page fault for address: ffff9e949b92d359
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305114] #PF: supervisor read access in kernel mode
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305114] #PF: error_code(0x0000) - not-present page

--dump_pagetable

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305115] PGD 136a01067 P4D 136a01067 PUD 0 

-__die

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305117] Oops: 0000 [#1] SMP PTI

--show_regs

---show_regs_print_info

----dump_stack_print_info

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305119] CPU: 1 PID: 5234 Comm: bash Tainted: G           OE     5.4.0 #1
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305120] Hardware name: Dell Inc. Inspiron 7590/08717F, BIOS 1.7.0 06/30/2020

---__show_regs

----show_iret_regs

-----show_ip

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305124] RIP: 0010:__task_pid_nr_ns+0x7f/0x90

------show_opcodes

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305126] Code: eb d6 65 48 8b 04 25 c0 6b 01 00 48 8b 88 28 09 00 00 48 85 c9 74 a2 8b 41 04 48 83 c0 05 48 c1 e0 04 48 83 bf 28 09 00 00 00 <48> 8b 54 01 08 75 92 31 c0 5d c3 8b 41 50 5d c3 90 0f 1f 44 00 00

----show_iret_regs #back

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305126] RSP: 0018:ffffb500c2c67f18 EFLAGS: 00010282

---__show_regs #back

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305127] RAX: 0000000b00000050 RBX: 0000000000000000 RCX: ffff9e899b92d301
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305128] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9e8984532f80
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305129] RBP: ffffb500c2c67f18 R08: 0000000000000000 R09: 0000000000000000
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305129] R10: 0000000000000000 R11: 0000000000000000 R12: ffffb500c2c67f58
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305130] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305131] FS:  00007ff19993b740(0000) GS:ffff9e89ad480000(0000) knlGS:0000000000000000
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305132] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305133] CR2: ffff9e949b92d359 CR3: 000000045fc1a003 CR4: 00000000001606e0

---show_trace_log_lvl #if (!user_mode(regs))

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305133] Call Trace:
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305137]  __ia32_sys_getpid+0x1e/0x30
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305140]  do_syscall_64+0x57/0x190
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305143]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305144] RIP: 0033:0x7ff198ff1937
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305145] Code: e9 49 ff ff ff 44 89 f2 41 0f b7 4c 17 fe 66 89 4c 10 fe e9 36 ff ff ff e8 e6 f1 04 00 66 0f 1f 44 00 00 b8 27 00 00 00 0f 05 <c3> 0f 1f 84 00 00 00 00 00 b8 6e 00 00 00 0f 05 c3 0f 1f 84 00 00
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305146] RSP: 002b:00007fff7278fd58 EFLAGS: 00000246 ORIG_RAX: 0000000000000027
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305147] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff198ff1937
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305148] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305148] RBP: 0000000000000001 R08: 00007ff19993b740 R09: 0000000000000000
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305149] R10: 00007ff19993ba10 R11: 0000000000000246 R12: 00007fff7278fd80
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305149] R13: 00007fff7278fe00 R14: 0000560cfa2bef10 R15: 0000000000000000

--print_modules

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305150] Modules linked in: nuke(OE) rfcomm thunderbolt ccm cmac bnep snd_sof_pci snd_sof_intel_hda_common snd_soc_hdac_hda snd_sof_intel_hda x86_pkg_temp_thermal intel_powerclamp snd_sof_intel_byt snd_sof_intel_ipc snd_sof coretemp snd_sof_xtensa_dsp snd_hda_ext_core kvm_intel snd_soc_acpi_intel_match kvm snd_hda_codec_hdmi snd_soc_acpi mei_hdcp snd_soc_core hid_multitouch snd_hda_codec_realtek crct10dif_pclmul snd_hda_codec_generic snd_compress ac97_bus crc32_pclmul snd_pcm_dmaengine snd_hda_intel intel_rapl_msr ghash_clmulni_intel iwlmvm snd_intel_nhlt snd_hda_codec dell_laptop ledtrig_audio snd_hda_core mac80211 snd_hwdep snd_pcm dell_smm_hwmon snd_seq_midi snd_seq_midi_event aesni_intel crypto_simd cryptd glue_helper snd_rawmidi intel_cstate libarc4 nls_iso8859_1 intel_rapl_perf dell_wmi serio_raw dell_smbios dcdbas iwlwifi snd_seq snd_seq_device snd_timer i915 snd cfg80211 nouveau uvcvideo dell_wmi_descriptor alienware_wmi wmi_bmof intel_wmi_thunderbolt soundcore videobuf2_vmalloc
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305175]  videobuf2_memops videobuf2_v4l2 btusb videobuf2_common btrtl videodev btbcm mei_me btintel mei bluetooth mc mxm_wmi input_leds ttm ucsi_acpi typec_ucsi ecdh_generic ecc joydev drm_kms_helper typec mac_hid drm int3403_thermal i2c_algo_bit fb_sys_fops syscopyarea sysfillrect processor_thermal_device sysimgblt intel_lpss_pci int3400_thermal intel_rapl_common intel_hid intel_lpss int340x_thermal_zone acpi_pad acpi_thermal_rel idma64 sparse_keymap intel_soc_dts_iosf cros_ec_ishtp virt_dma cros_ec intel_pch_thermal sch_fq_codel isgx(OE) parport_pc ppdev lp parport ip_tables x_tables autofs4 usbhid hid_sensor_custom hid_sensor_hub hid_generic intel_ishtp_loader intel_ishtp_hid nvme nvme_core intel_ish_ipc ahci intel_ishtp libahci i2c_hid hid wmi video pinctrl_cannonlake pinctrl_intel

no_context #back

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305200] CR2: ffff9e949b92d359

-oops_end

--oops_exit

---print_oops_end_marker

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305202] ---[ end trace 484fd115e18dfbaf ]---

--__show_regs

Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305203] RIP: 0010:__task_pid_nr_ns+0x7f/0x90
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305204] Code: eb d6 65 48 8b 04 25 c0 6b 01 00 48 8b 88 28 09 00 00 48 85 c9 74 a2 8b 41 04 48 83 c0 05 48 c1 e0 04 48 83 bf 28 09 00 00 00 <48> 8b 54 01 08 75 92 31 c0 5d c3 8b 41 50 5d c3 90 0f 1f 44 00 00
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305205] RSP: 0018:ffffb500c2c67f18 EFLAGS: 00010282
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305206] RAX: 0000000b00000050 RBX: 0000000000000000 RCX: ffff9e899b92d301
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305207] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9e8984532f80
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305207] RBP: ffffb500c2c67f18 R08: 0000000000000000 R09: 0000000000000000
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305208] R10: 0000000000000000 R11: 0000000000000000 R12: ffffb500c2c67f58
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305209] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305209] FS:  00007ff19993b740(0000) GS:ffff9e89ad480000(0000) knlGS:0000000000000000
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305210] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 11 21:18:47 leone-Inspiron-7590 kernel: [  917.305211] CR2: ffff9e949b92d359 CR3: 000000045fc1a003 CR4: 00000000001606e0

OOPS官方文档

linux-5.4/Documentation/translations/zh_CN/oops-tracing.txt

Chinese translated version of Documentation/admin-guide/bug-hunting.rst

If you have any comment or update to the content, please contact the
original document maintainer directly.  However, if you have a problem
communicating in English you can also ask the Chinese maintainer for
help.  Contact the Chinese maintainer if this translation is outdated
or if there is a problem with the translation.

Chinese maintainer: Dave Young <hidave.darkstar@gmail.com>
---------------------------------------------------------------------
Documentation/admin-guide/bug-hunting.rst 的中文翻译

如果想评论或更新本文的内容，请直接联系原文档的维护者。如果你使用英文
交流有困难的话，也可以向中文版维护者求助。如果本翻译更新不及时或者翻
译存在问题，请联系中文版维护者。

中文版维护者： 杨瑞 Dave Young <hidave.darkstar@gmail.com>
中文版翻译者： 杨瑞 Dave Young <hidave.darkstar@gmail.com>
中文版校译者： 李阳 Li Yang <leoyang.li@nxp.com>
               王聪 Wang Cong <xiyou.wangcong@gmail.com>

以下为正文
---------------------------------------------------------------------

注意： ksymoops 在2.6中是没有用的。 请以原有格式使用Oops(来自dmesg，等等)。
忽略任何这样那样关于“解码Oops”或者“通过ksymoops运行”的文档。 如果你贴出运行过
ksymoops的来自2.6的Oops，人们只会让你重贴一次。

快速总结
-------------

发现Oops并发送给看似相关的内核领域的维护者。别太担心对不上号。如果你不确定就发给
和你所做的事情相关的代码的负责人。 如果可重现试着描述怎样重构。 那甚至比oops更有
价值。

如果你对于发送给谁一无所知， 发给linux-kernel@vger.kernel.org。感谢你帮助Linux
尽可能地稳定。

Oops在哪里?
----------------------

通常Oops文本由klogd从内核缓冲区里读取并传给syslogd，由syslogd写到syslog文件中，
典型地是/var/log/messages(依赖于/etc/syslog.conf)。有时klogd崩溃了,这种情况下你
能够运行dmesg > file来从内核缓冲区中读取数据并保存下来。 否则你可以
cat /proc/kmsg > file， 然而你必须介入中止传输， kmsg是一个“永不结束的文件”。如
果机器崩溃坏到你不能输入命令或者磁盘不可用那么你有三种选择:-

（1） 手抄屏幕上的文本待机器重启后再输入计算机。 麻烦但如果没有针对崩溃的准备，
这是仅有的选择。 另外，你可以用数码相机把屏幕拍下来-不太好，但比没有强。 如果信
息滚动到了终端的上面，你会发现以高分辩率启动（比如，vga=791）会让你读到更多的文
本。（注意：这需要vesafb，所以对‘早期’的oops没有帮助）

（2）用串口终端启动（请参看Documentation/admin-guide/serial-console.rst），运行一个null
modem到另一台机器并用你喜欢的通讯工具获取输出。Minicom工作地很好。

（3）使用Kdump（请参看Documentation/admin-guide/kdump/kdump.rst），
使用在Documentation/admin-guide/kdump/gdbmacros.txt中定义的dmesg gdb宏，从旧的内存中提取内核
环形缓冲区。

完整信息
----------------

注意：以下来自于Linus的邮件适用于2.4内核。 我因为历史原因保留了它，并且因为其中
一些信息仍然适用。 特别注意的是，请忽略任何ksymoops的引用。

From: Linus Torvalds <torvalds@osdl.org>

怎样跟踪Oops.. [原发到linux-kernel的一封邮件]

主要的窍门是有五年和这些烦人的oops消息打交道的经验;-)

实际上，你有办法使它更简单。我有两个不同的方法：

	gdb /usr/src/linux/vmlinux
	gdb> disassemble <offending_function>

那是发现问题的简单办法，至少如果bug报告做的好的情况下（象这个一样-运行ksymoops
得到oops发生的函数及函数内的偏移）。

哦，如果报告发生的内核以相同的编译器和相似的配置编译它会有帮助的。

另一件要做的事是反汇编bug报告的“Code”部分：ksymoops也会用正确的工具来做这件事，
但如果没有那些工具你可以写一个傻程序：

	char str[] = "\xXX\xXX\xXX...";
	main(){}

并用gcc -g编译它然后执行“disassemble str”（XX部分是由Oops报告的值-你可以仅剪切
粘贴并用“\x”替换空格-我就是这么做的，因为我懒得写程序自动做这一切）。

另外，你可以用scripts/decodecode这个shell脚本。它的使用方法是：
decodecode < oops.txt

“Code”之后的十六进制字节可能（在某些架构上）有一些当前指令之前的指令字节以及
当前和之后的指令字节

Code: f9 0f 8d f9 00 00 00 8d 42 0c e8 dd 26 11 c7 a1 60 ea 2b f9 8b 50 08 a1
64 ea 2b f9 8d 34 82 8b 1e 85 db 74 6d 8b 15 60 ea 2b f9 <8b> 43 04 39 42 54
7e 04 40 89 42 54 8b 43 04 3b 05 00 f6 52 c0

最后，如果你想知道代码来自哪里，你可以：

	cd /usr/src/linux
	make fs/buffer.s 	# 或任何产生BUG的文件

然后你会比gdb反汇编更清楚的知道发生了什么。

现在，问题是把你所拥有的所有数据结合起来：C源码（关于它应该怎样的一般知识），
汇编代码及其反汇编得到的代码（另外还有从“oops”消息得到的寄存器状态-对了解毁坏的
指针有用，而且当你有了汇编代码你也能拿其它的寄存器和任何它们对应的C表达式做匹配
）。

实际上，你仅需看看哪里不匹配（这个例子是“Code”反汇编和编译器生成的代码不匹配）。
然后你须要找出为什么不匹配。通常很简单-你看到代码使用了空指针然后你看代码想知道
空指针是怎么出现的，还有检查它是否合法..

现在，如果明白这是一项耗时的工作而且需要一丁点儿的专心，没错。这就是我为什么大多
只是忽略那些没有符号表信息的崩溃报告的原因：简单的说太难查找了（我有一些
程序用于在内核代码段中搜索特定的模式，而且有时我也已经能找出那些崩溃的地方，但是
仅仅是找出正确的序列也确实需要相当扎实的内核知识）

_有时_会发生这种情况，我仅看到崩溃中的反汇编代码序列， 然后我马上就明白问题出在
哪里。这时我才意识到自己干这个工作已经太长时间了;-)

		Linus


---------------------------------------------------------------------------
关于Oops跟踪的注解：

为了帮助Linus和其它内核开发者，klogd纳入了大量的支持来处理保护错误。为了拥有对
地址解析的完整支持至少应该使用1.3-pl3的sysklogd包。

当保护错误发生时，klogd守护进程自动把内核日志信息中的重要地址翻译成它们相应的符
号。

klogd执行两种类型的地址解析。首先是静态翻译其次是动态翻译。静态翻译和ksymoops
一样使用System.map文件。为了做静态翻译klogd守护进程必须在初始化时能找到system
map文件。关于klogd怎样搜索map文件请参看klogd手册页。

动态地址翻译在使用内核可装载模块时很重要。 因为内核模块的内存是从内核动态内存池
里分配的，所以不管是模块开始位置还是模块中函数和符号的位置都不是固定的。

内核支持允许程序决定装载哪些模块和它们在内存中位置的系统调用。使用这些系统调用
klogd守护进程生成一张符号表用于调试发生在可装载模块中的保护错误。

至少klogd会提供产生保护错误的模块名。还可有额外的符号信息供可装载模块开发者选择
以从模块中输出符号信息。

因为内核模块环境可能是动态的，所以必须有一种机制当模块环境发生改变时来通知klogd
守护进程。 有一些可用的命令行选项允许klogd向当前执行中的守护进程发送信号，告知符
号信息应该被刷新了。 更多信息请参看klogd手册页。

sysklogd发布时包含一个补丁修改了modules-2.0.0包，无论何时一个模块装载或者卸载都
会自动向klogd发送信号。打上这个补丁提供了必要的对调试发生于内核可装载模块的保护
错误的无缝支持。

以下是被klogd处理过的发生在可装载模块中的一个保护错误例子：
---------------------------------------------------------------------------
Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc
Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000
Aug 29 09:51:01 blizard kernel: *pde = 00000000
Aug 29 09:51:01 blizard kernel: Oops: 0002
Aug 29 09:51:01 blizard kernel: CPU:    0
Aug 29 09:51:01 blizard kernel: EIP:    0010:[oops:_oops+16/3868]
Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212
Aug 29 09:51:01 blizard kernel: eax: 315e97cc   ebx: 003a6f80   ecx: 001be77b   edx: 00237c0c
Aug 29 09:51:01 blizard kernel: esi: 00000000   edi: bffffdb3   ebp: 00589f90   esp: 00589f8c
Aug 29 09:51:01 blizard kernel: ds: 0018   es: 0018   fs: 002b   gs: 002b   ss: 0018
Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000)
Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001
Aug 29 09:51:01 blizard kernel:        00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00
Aug 29 09:51:01 blizard kernel:        bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036
Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128]
Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3
---------------------------------------------------------------------------

Dr. G.W. Wettstein           Oncology Research Div. Computing Facility
Roger Maris Cancer Center    INTERNET: greg@wind.rmcc.com
820 4th St. N.
Fargo, ND  58122
Phone: 701-234-7556


---------------------------------------------------------------------------
受污染的内核

一些oops报告在程序记数器之后包含字符串'Tainted: '。这表明内核已经被一些东西给污
染了。 该字符串之后紧跟着一系列的位置敏感的字符，每个代表一个特定的污染值。

  1：'G'如果所有装载的模块都有GPL或相容的许可证，'P'如果装载了任何的专有模块。
没有模块MODULE_LICENSE或者带有insmod认为是与GPL不相容的的MODULE_LICENSE的模块被
认定是专有的。

  2：'F'如果有任何通过“insmod -f”被强制装载的模块，' '如果所有模块都被正常装载。

  3：'S'如果oops发生在SMP内核中，运行于没有证明安全运行多处理器的硬件。 当前这种
情况仅限于几种不支持SMP的速龙处理器。

  4：'R'如果模块通过“insmod -f”被强制装载，' '如果所有模块都被正常装载。

  5：'M'如果任何处理器报告了机器检查异常，' '如果没有发生机器检查异常。

  6：'B'如果页释放函数发现了一个错误的页引用或者一些非预期的页标志。

  7：'U'如果用户或者用户应用程序特别请求设置污染标志，否则' '。

  8：'D'如果内核刚刚死掉，比如有OOPS或者BUG。

使用'Tainted: '字符串的主要原因是要告诉内核调试者，这是否是一个干净的内核亦或发
生了任何的不正常的事。污染是永久的：即使出错的模块已经被卸载了，污染值仍然存在，
以表明内核不再值得信任。

linux-5.4/Documentation/admin-guide/bug-hunting.rst 

Bug hunting
===========

Kernel bug reports often come with a stack dump like the one below::

	------------[ cut here ]------------
	WARNING: CPU: 1 PID: 28102 at kernel/module.c:1108 module_put+0x57/0x70
	Modules linked in: dvb_usb_gp8psk(-) dvb_usb dvb_core nvidia_drm(PO) nvidia_modeset(PO) snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hwdep snd_hda_core snd_pcm snd_timer snd soundcore nvidia(PO) [last unloaded: rc_core]
	CPU: 1 PID: 28102 Comm: rmmod Tainted: P        WC O 4.8.4-build.1 #1
	Hardware name: MSI MS-7309/MS-7309, BIOS V1.12 02/23/2009
	 00000000 c12ba080 00000000 00000000 c103ed6a c1616014 00000001 00006dc6
	 c1615862 00000454 c109e8a7 c109e8a7 00000009 ffffffff 00000000 f13f6a10
	 f5f5a600 c103ee33 00000009 00000000 00000000 c109e8a7 f80ca4d0 c109f617
	Call Trace:
	 [<c12ba080>] ? dump_stack+0x44/0x64
	 [<c103ed6a>] ? __warn+0xfa/0x120
	 [<c109e8a7>] ? module_put+0x57/0x70
	 [<c109e8a7>] ? module_put+0x57/0x70
	 [<c103ee33>] ? warn_slowpath_null+0x23/0x30
	 [<c109e8a7>] ? module_put+0x57/0x70
	 [<f80ca4d0>] ? gp8psk_fe_set_frontend+0x460/0x460 [dvb_usb_gp8psk]
	 [<c109f617>] ? symbol_put_addr+0x27/0x50
	 [<f80bc9ca>] ? dvb_usb_adapter_frontend_exit+0x3a/0x70 [dvb_usb]
	 [<f80bb3bf>] ? dvb_usb_exit+0x2f/0xd0 [dvb_usb]
	 [<c13d03bc>] ? usb_disable_endpoint+0x7c/0xb0
	 [<f80bb48a>] ? dvb_usb_device_exit+0x2a/0x50 [dvb_usb]
	 [<c13d2882>] ? usb_unbind_interface+0x62/0x250
	 [<c136b514>] ? __pm_runtime_idle+0x44/0x70
	 [<c13620d8>] ? __device_release_driver+0x78/0x120
	 [<c1362907>] ? driver_detach+0x87/0x90
	 [<c1361c48>] ? bus_remove_driver+0x38/0x90
	 [<c13d1c18>] ? usb_deregister+0x58/0xb0
	 [<c109fbb0>] ? SyS_delete_module+0x130/0x1f0
	 [<c1055654>] ? task_work_run+0x64/0x80
	 [<c1000fa5>] ? exit_to_usermode_loop+0x85/0x90
	 [<c10013f0>] ? do_fast_syscall_32+0x80/0x130
	 [<c1549f43>] ? sysenter_past_esp+0x40/0x6a
	---[ end trace 6ebc60ef3981792f ]---

Such stack traces provide enough information to identify the line inside the
Kernel's source code where the bug happened. Depending on the severity of
the issue, it may also contain the word **Oops**, as on this one::

	BUG: unable to handle kernel NULL pointer dereference at   (null)
	IP: [<c06969d4>] iret_exc+0x7d0/0xa59
	*pdpt = 000000002258a001 *pde = 0000000000000000
	Oops: 0002 [#1] PREEMPT SMP
	...

Despite being an **Oops** or some other sort of stack trace, the offended
line is usually required to identify and handle the bug. Along this chapter,
we'll refer to "Oops" for all kinds of stack traces that need to be analized.

.. note::

  ``ksymoops`` is useless on 2.6 or upper.  Please use the Oops in its original
  format (from ``dmesg``, etc).  Ignore any references in this or other docs to
  "decoding the Oops" or "running it through ksymoops".
  If you post an Oops from 2.6+ that has been run through ``ksymoops``,
  people will just tell you to repost it.

Where is the Oops message is located?
-------------------------------------

Normally the Oops text is read from the kernel buffers by klogd and
handed to ``syslogd`` which writes it to a syslog file, typically
``/var/log/messages`` (depends on ``/etc/syslog.conf``). On systems with
systemd, it may also be stored by the ``journald`` daemon, and accessed
by running ``journalctl`` command.

Sometimes ``klogd`` dies, in which case you can run ``dmesg > file`` to
read the data from the kernel buffers and save it.  Or you can
``cat /proc/kmsg > file``, however you have to break in to stop the transfer,
``kmsg`` is a "never ending file".

If the machine has crashed so badly that you cannot enter commands or
the disk is not available then you have three options:

(1) Hand copy the text from the screen and type it in after the machine
    has restarted.  Messy but it is the only option if you have not
    planned for a crash. Alternatively, you can take a picture of
    the screen with a digital camera - not nice, but better than
    nothing.  If the messages scroll off the top of the console, you
    may find that booting with a higher resolution (eg, ``vga=791``)
    will allow you to read more of the text. (Caveat: This needs ``vesafb``,
    so won't help for 'early' oopses)

(2) Boot with a serial console (see
    :ref:`Documentation/admin-guide/serial-console.rst <serial_console>`),
    run a null modem to a second machine and capture the output there
    using your favourite communication program.  Minicom works well.

(3) Use Kdump (see Documentation/admin-guide/kdump/kdump.rst),
    extract the kernel ring buffer from old memory with using dmesg
    gdbmacro in Documentation/admin-guide/kdump/gdbmacros.txt.

Finding the bug's location
--------------------------

Reporting a bug works best if you point the location of the bug at the
Kernel source file. There are two methods for doing that. Usually, using
``gdb`` is easier, but the Kernel should be pre-compiled with debug info.

gdb
^^^

The GNU debug (``gdb``) is the best way to figure out the exact file and line
number of the OOPS from the ``vmlinux`` file.

The usage of gdb works best on a kernel compiled with ``CONFIG_DEBUG_INFO``.
This can be set by running::

  $ ./scripts/config -d COMPILE_TEST -e DEBUG_KERNEL -e DEBUG_INFO

On a kernel compiled with ``CONFIG_DEBUG_INFO``, you can simply copy the
EIP value from the OOPS::

 EIP:    0060:[<c021e50e>]    Not tainted VLI

And use GDB to translate that to human-readable form::

  $ gdb vmlinux
  (gdb) l *0xc021e50e

If you don't have ``CONFIG_DEBUG_INFO`` enabled, you use the function
offset from the OOPS::

 EIP is at vt_ioctl+0xda8/0x1482

And recompile the kernel with ``CONFIG_DEBUG_INFO`` enabled::

  $ ./scripts/config -d COMPILE_TEST -e DEBUG_KERNEL -e DEBUG_INFO
  $ make vmlinux
  $ gdb vmlinux
  (gdb) l *vt_ioctl+0xda8
  0x1888 is in vt_ioctl (drivers/tty/vt/vt_ioctl.c:293).
  288	{
  289		struct vc_data *vc = NULL;
  290		int ret = 0;
  291
  292		console_lock();
  293		if (VT_BUSY(vc_num))
  294			ret = -EBUSY;
  295		else if (vc_num)
  296			vc = vc_deallocate(vc_num);
  297		console_unlock();

or, if you want to be more verbose::

  (gdb) p vt_ioctl
  $1 = {int (struct tty_struct *, unsigned int, unsigned long)} 0xae0 <vt_ioctl>
  (gdb) l *0xae0+0xda8

You could, instead, use the object file::

  $ make drivers/tty/
  $ gdb drivers/tty/vt/vt_ioctl.o
  (gdb) l *vt_ioctl+0xda8

If you have a call trace, such as::

     Call Trace:
      [<ffffffff8802c8e9>] :jbd:log_wait_commit+0xa3/0xf5
      [<ffffffff810482d9>] autoremove_wake_function+0x0/0x2e
      [<ffffffff8802770b>] :jbd:journal_stop+0x1be/0x1ee
      ...

this shows the problem likely in the :jbd: module. You can load that module
in gdb and list the relevant code::

  $ gdb fs/jbd/jbd.ko
  (gdb) l *log_wait_commit+0xa3

.. note::

     You can also do the same for any function call at the stack trace,
     like this one::

	 [<f80bc9ca>] ? dvb_usb_adapter_frontend_exit+0x3a/0x70 [dvb_usb]

     The position where the above call happened can be seen with::

	$ gdb drivers/media/usb/dvb-usb/dvb-usb.o
	(gdb) l *dvb_usb_adapter_frontend_exit+0x3a

objdump
^^^^^^^

To debug a kernel, use objdump and look for the hex offset from the crash
output to find the valid line of code/assembler. Without debug symbols, you
will see the assembler code for the routine shown, but if your kernel has
debug symbols the C code will also be available. (Debug symbols can be enabled
in the kernel hacking menu of the menu configuration.) For example::

    $ objdump -r -S -l --disassemble net/dccp/ipv4.o

.. note::

   You need to be at the top level of the kernel tree for this to pick up
   your C files.

If you don't have access to the code you can also debug on some crash dumps
e.g. crash dump output as shown by Dave Miller::

     EIP is at 	+0x14/0x4c0
      ...
     Code: 44 24 04 e8 6f 05 00 00 e9 e8 fe ff ff 8d 76 00 8d bc 27 00 00
     00 00 55 57  56 53 81 ec bc 00 00 00 8b ac 24 d0 00 00 00 8b 5d 08
     <8b> 83 3c 01 00 00 89 44  24 14 8b 45 28 85 c0 89 44 24 18 0f 85

     Put the bytes into a "foo.s" file like this:

            .text
            .globl foo
     foo:
            .byte  .... /* bytes from Code: part of OOPS dump */

     Compile it with "gcc -c -o foo.o foo.s" then look at the output of
     "objdump --disassemble foo.o".

     Output:

     ip_queue_xmit:
         push       %ebp
         push       %edi
         push       %esi
         push       %ebx
         sub        $0xbc, %esp
         mov        0xd0(%esp), %ebp        ! %ebp = arg0 (skb)
         mov        0x8(%ebp), %ebx         ! %ebx = skb->sk
         mov        0x13c(%ebx), %eax       ! %eax = inet_sk(sk)->opt

Reporting the bug
-----------------

Once you find where the bug happened, by inspecting its location,
you could either try to fix it yourself or report it upstream.

In order to report it upstream, you should identify the mailing list
used for the development of the affected code. This can be done by using
the ``get_maintainer.pl`` script.

For example, if you find a bug at the gspca's sonixj.c file, you can get
their maintainers with::

	$ ./scripts/get_maintainer.pl -f drivers/media/usb/gspca/sonixj.c
	Hans Verkuil <hverkuil@xs4all.nl> (odd fixer:GSPCA USB WEBCAM DRIVER,commit_signer:1/1=100%)
	Mauro Carvalho Chehab <mchehab@kernel.org> (maintainer:MEDIA INPUT INFRASTRUCTURE (V4L/DVB),commit_signer:1/1=100%)
	Tejun Heo <tj@kernel.org> (commit_signer:1/1=100%)
	Bhaktipriya Shridhar <bhaktipriya96@gmail.com> (commit_signer:1/1=100%,authored:1/1=100%,added_lines:4/4=100%,removed_lines:9/9=100%)
	linux-media@vger.kernel.org (open list:GSPCA USB WEBCAM DRIVER)
	linux-kernel@vger.kernel.org (open list)

Please notice that it will point to:

- The last developers that touched on the source code. On the above example,
  Tejun and Bhaktipriya (in this specific case, none really envolved on the
  development of this file);
- The driver maintainer (Hans Verkuil);
- The subsystem maintainer (Mauro Carvalho Chehab);
- The driver and/or subsystem mailing list (linux-media@vger.kernel.org);
- the Linux Kernel mailing list (linux-kernel@vger.kernel.org).

Usually, the fastest way to have your bug fixed is to report it to mailing
list used for the development of the code (linux-media ML) copying the driver maintainer (Hans).

If you are totally stumped as to whom to send the report, and
``get_maintainer.pl`` didn't provide you anything useful, send it to
linux-kernel@vger.kernel.org.

Thanks for your help in making Linux as stable as humanly possible.

Fixing the bug
--------------

If you know programming, you could help us by not only reporting the bug,
but also providing us with a solution. After all, open source is about
sharing what you do and don't you want to be recognised for your genius?

If you decide to take this way, once you have worked out a fix please submit
it upstream.

Please do read
:ref:`Documentation/process/submitting-patches.rst <submittingpatches>` though
to help your code get accepted.


---------------------------------------------------------------------------

Notes on Oops tracing with ``klogd``
------------------------------------

In order to help Linus and the other kernel developers there has been
substantial support incorporated into ``klogd`` for processing protection
faults.  In order to have full support for address resolution at least
version 1.3-pl3 of the ``sysklogd`` package should be used.

When a protection fault occurs the ``klogd`` daemon automatically
translates important addresses in the kernel log messages to their
symbolic equivalents.  This translated kernel message is then
forwarded through whatever reporting mechanism ``klogd`` is using.  The
protection fault message can be simply cut out of the message files
and forwarded to the kernel developers.

Two types of address resolution are performed by ``klogd``.  The first is
static translation and the second is dynamic translation.  Static
translation uses the System.map file in much the same manner that
ksymoops does.  In order to do static translation the ``klogd`` daemon
must be able to find a system map file at daemon initialization time.
See the klogd man page for information on how ``klogd`` searches for map
files.

Dynamic address translation is important when kernel loadable modules
are being used.  Since memory for kernel modules is allocated from the
kernel's dynamic memory pools there are no fixed locations for either
the start of the module or for functions and symbols in the module.

The kernel supports system calls which allow a program to determine
which modules are loaded and their location in memory.  Using these
system calls the klogd daemon builds a symbol table which can be used
to debug a protection fault which occurs in a loadable kernel module.

At the very minimum klogd will provide the name of the module which
generated the protection fault.  There may be additional symbolic
information available if the developer of the loadable module chose to
export symbol information from the module.

Since the kernel module environment can be dynamic there must be a
mechanism for notifying the ``klogd`` daemon when a change in module
environment occurs.  There are command line options available which
allow klogd to signal the currently executing daemon that symbol
information should be refreshed.  See the ``klogd`` manual page for more
information.

A patch is included with the sysklogd distribution which modifies the
``modules-2.0.0`` package to automatically signal klogd whenever a module
is loaded or unloaded.  Applying this patch provides essentially
seamless support for debugging protection faults which occur with
kernel loadable modules.

The following is an example of a protection fault in a loadable module
processed by ``klogd``::

	Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc
	Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000
	Aug 29 09:51:01 blizard kernel: *pde = 00000000
	Aug 29 09:51:01 blizard kernel: Oops: 0002
	Aug 29 09:51:01 blizard kernel: CPU:    0
	Aug 29 09:51:01 blizard kernel: EIP:    0010:[oops:_oops+16/3868]
	Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212
	Aug 29 09:51:01 blizard kernel: eax: 315e97cc   ebx: 003a6f80   ecx: 001be77b   edx: 00237c0c
	Aug 29 09:51:01 blizard kernel: esi: 00000000   edi: bffffdb3   ebp: 00589f90   esp: 00589f8c
	Aug 29 09:51:01 blizard kernel: ds: 0018   es: 0018   fs: 002b   gs: 002b   ss: 0018
	Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000)
	Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001
	Aug 29 09:51:01 blizard kernel:        00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00
	Aug 29 09:51:01 blizard kernel:        bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036
	Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128]
	Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3

---------------------------------------------------------------------------

::

  Dr. G.W. Wettstein           Oncology Research Div. Computing Facility
  Roger Maris Cancer Center    INTERNET: greg@wind.rmcc.com
  820 4th St. N.
  Fargo, ND  58122
  Phone: 701-234-7556