使用kdb和kgdb调试Linux内核(3)

This article was firstly published from http://oliveryang.net. The content reuse need include the original link.

1. Debug kernel modules by kgdb

In Linux kdb, kgdb, gdb - 2, we knew how to use gdb as client to debug remote Linux kernel via kgdb.
It would be no problems, if our debug target is kernel. Setting break points in kernel for live control is quite straightforward. However, if the debug target is existing in a separate kernel module or driver, it may need additional tricks. This article covers the major tricks to debug a kernel module by kgdb.

Using kgdb has two major problems on kernel module debug,

• The gdb client needs to access the debug version of module binaries.

  This requires to install the module debug binaries on gdb client side first. For the module built by ourself, it should be quite easy. While debugging on some commercial Linux distributions, the default module binaries are non-debug version. We have to follow the method provided by vendors to get the debug version. There is no standard methods here, different vendors have different methods. For example, RHEL/CentOS/Fedora uses debuginfo-install command to download and install the debug binaries. Whereas, Ubuntu just provides a website for debug binaries download and manual install. Anyway, most of Linux distributions don’t install the debug binaries by default. Please follow the methods provided by vendors to install debug binaries.

• The gdb client also needs to know kernel module load address.

  Otherwise, gdb couldn’t resolve module symbol address specified in gdb debug instructions. The add-symbol-file command could be used here. The command accepts the install path of the kernel module debug binaries. It also requires following module ELF section addresses for different debug purposes,

  • .text section address for major module function symbols, which is mandated.
  • .init.text section address for module init function symbols, which is optional.
  • .exit.text section address for module exit function symbols, which is optional.
  • .data section address for global variable which is optional.
  • .bss section address for static variable which is optional.

  Linux kernel modules follow the ELF specification. The add-symbol-file command accepts all these sections base addresses. We just cover most popular sections here. For the other sections, you may use them for other special purposes. Before using the command, user should have clear concepts about which sections base addresses are required for the debug purposes.

After addressing above two problems, the remaining things are mainly module specific. User needs to have enough knowledge regarding to the module they want to debug.

This article gives some real examples on how to use add-symbol-file command properly in different debug scenarios. In these examples, two machine got connected via serial cable or serial console. The gdb client from one machine could connect to another machine if another machine dropped into kgdb debugger. And gdb client already can set break points in kernel(vmlinux), and control the kernel execution if we want.

Anyway, at this time point, you couldn’t set break points in kernel modules or drivers without running add-symbol-file command correctly. If you still have the problems on connecting to kgdb via gdb client, please refer to Using kdb/kgdb debug Linux kernel - 2.

2. Module debugging after system boot

This debug scenario assumes that the kernel module need to be debugged after Linux OS already boot successfully.

Linux provides all kernel module section names and addresses from sysfs interfaces: /sys/module/[module name]/sections/. By leveraging sysfs interfaces, the module ELF sections base addresses could be very easily acquired.

Here is an example of debugging Intel e1000 driver after system boot. Our goal is,

Setting break points in e1000 driver interrupt handler and checking
driver key data structures.

In fact, all my debug steps were done on my two VMs, which got connected by virtual serial console. The OS version is CentOS 7.2 with debug kernel modules installed.

2.1 Loading e1000 driver symbols

• First, get module ELF sections base addresses

  As system is still functional, the e1000 module ELF section address could be got by sysfs interfaces. Per our debug goals, we just need to focus on follow ELF sections,

  cat/sys/module/e1000/sectio<