Debugging Kernel with KGDB

337人阅读 评论(0) 收藏 举报


The Problem

Suppose you’ve written kernel module and it do not work as intended. You cannot find anything by reading code. printk debugging left you with noting. You wish there was a way to look how things act in the wild… It would be nice to run some debugger, create few breakpoints and operate on live data, just like you used to do with your desktop applications…

As it appears, some twisted fairy already fulfilled that wish. The thing is called KGDB and it is both a little scary and pretty awesome.


KGDB, as it’s name suggests, uses GDB, and does that in all it’s glory. You can peek at variables, execute code, jump around, create breakpoints (even conditional ones) peek at threads, etc. All this while having corresponding source in front of you.

Luckily for us KGDB in a light version was merged around mid-2.6 series into mainline. That means no patching. We just must have kernel with few options turned on (in most distributions, if not all, it means recompilation) and need serial port connection between two PCs (virtual machine with emulated RS232 is fine).


You need to build kernel on target machine with following options:


Note that those are essentials and there are lots of other DEBUG options. They are in form of:


Check them out in Kernel Hacking configuration section.

If you want to step into not your code you should also turn off GCC optimizations (changing -O2 flag to -O0) on parts of the kernel of your interest. Otherwise be prepared to strange behaviour when viewing code. There will be optimized out values, odd code jumping, etc. Your module also should be compiled with -O0, -g and -ggdb options to add in symbols and turn off optimizations.

Remember - do not ever turn optimizations on the whole kernel. Developers used some wicked wizardry like depending on function in-lining that comes with -O2 flag. Out in the intertubes exists patch that aims stripping optimizations from most of the Linux, excluding parts that needs them. You may be interested, I haven’t found it usable.


On the other end of serial connection, in order to run GDB, you will need vmlinux file (it is uncompressed and contains debugging symbols) which you should find in root of kernel build directory. The file should weight around 100MiB. Additionally you have to keep kernel and module sources around since it will be handy to have GDB print them along.


If you physically connected two machines then you are ready with ttyS0. If you used VM’s virtual COM then you should get socket somewhere in file system. Check configuration for specific path. You can use socat to turn it into char device.

socat -d -d /home/ivyl/virtualbox/myvm/serial1 pty

It will print out device path it created.

Target Machine

We need to prepare target (the one to be debugged) system. KGDB needs to know on which device it’s supposed to listen and which baudrate it should use. You can do it via /sys:

echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc

or by adding kernel parameter:


If you are using virutalized com ttyS0 should be fine.

You may also be interested in kgdbwait parameter. It will make kernel to break as soon as possible during boot process.

Now only thing left is to break the machine (stop execution, wait for debugger interaction) by pressing SysRq-G (where SysRq is Alt + PrtSc) or executing from command line the following:

echo g > /proc/sysrq-trigger

Firing GDB

With above preparations done, you have to attach debugger to it. I recommend doing it in kernel’s source directory. It’s easiest way of making GDB source aware.

cd /usr/src/linux-src
gdb /path/to/copied/vmlinux

Now you are within GDB shell. You can connect to remote machine by setting same baudrate and point device you want connect to.

set remotebaud 115200
target remote /dev/pts/4

You should substitude /dev/pts/4 with device of your choice (one created by socat or the physical).

If you broke target system you should be already able to poke around. If not, look above for advice how to do it.

If you don’t know what to do now take some example GDB tutorial. There is much more to GDB. You may also want to try GUI called DDD.

And Module?

That’s fine, but how to debug mentioned modules, you ask? Nothing simpler! If you are testing own module, in it’s build directory, after compilation with appropriate flags, there should be two files that matters to us. module.ko, which you will load through insmod, and module.o which we will load into GDB since it contains symbols.

Just load .o on the target machine. Now take a look at file:


It contains memory address where module was loaded. Now, on the other machine, you should feed GDB with module.o and mentioned address using:

add-symbol-file module.o 0xd80a4400

Voila! Now go debug like there’s no tomorrow.

PS. I also made this script to make repetitive task around KGDB automated. It’s not all pretty and shiny but maybe you will find it useful.


30 November 2012


ubuntu kgdb kernel 调试环境搭建

<!-- @page {margin:2cm} p {margin-bottom:0.21cm} a:link {color:#0000ff} --> 调试环境(末尾还有ubuntu下vmware的环境搭建方法): 主机:WindowsXP Profe...
  • dndxhej
  • dndxhej
  • 2012-01-02 19:38
  • 5943

imx6 KGDB调试方法总结(光谷王凯的博客)

1,平台环境 PC:ubuntu12.04 单板:imx6(android4.4.2) 2,imx6的kernel编译加gdb调试 make menuconfig,然后按照Kernelhacking-->KGDB: kernel debugger-->KGDB: use kg...
  • kao2406
  • kao2406
  • 2016-04-07 11:50
  • 1255

kernel debug

 调试是软件开发过程中一个必不可少的环节,在 Linux 内核开发的过程中也不可避免地会面对如何调试内核的问题。但是,Linux 系统的开发者出于保证内核代码正确性的考虑,不愿意在 Linux 内核源代码树中加入一个调试器。他们认为内核中的调试器会误导开发者,从而引入不良的修正[1].所以对 ...
  • prike
  • prike
  • 2016-07-30 17:32
  • 795


原理原理结构图原理过程Kgdb是双机在线调试,一端是Host端(linux),运行GDB,另一端是Target端,运行带Kgdb的linux内核。 两边通过串口(KGDBoc)或网络口(KGDBoE)相连接,kgdb实现了远程调试的功能,主要部件有:stubstub可卸任是一个运行在target端...
  • java211
  • java211
  • 2016-09-05 20:24
  • 875

linux内核调试gdb + KGDB

  • luckywang1103
  • luckywang1103
  • 2017-03-24 09:01
  • 554


从2.6.26开始,Linux 主干内核开始内置了代码级调试器 kgdb。通过 kgdb,可以在内核代码中设置断点,单步调试和观察变量。为了使用 kgdb,需要有两个系统。一个作为上位机,一个作为下位机(目标机)。两台机器通过串口线连接。需要调试的内核运行在下位机上。串口线用于kgdb连接远程目标板...
  • u013470224
  • u013470224
  • 2016-05-04 13:55
  • 1912


Linux内核可以远程用gdb调试。但调试内核某块还是有些特殊的问题要处理。本文主要介绍如何利用gdb kgdb 去调试Linux内核模块。
  • yayong
  • yayong
  • 2016-05-13 09:05
  • 2091

用KGdb和VMware调试Linux内核,System Call

Linux的内核和System Call不好调试,参考这里: http://stackoverflow.com/questions/5999205/cannot-step-into-system-call-source-code 简单来说,如果想在本机调试system call,那么当你进入...
  • hengyunabc
  • hengyunabc
  • 2014-05-03 20:59
  • 3402

Kernel low-level debugging functions linux汇编的调试方法

2011年7月份收藏的一片文章, 原标题是"Kernel low-level debugging functions linux汇编的调试方法". 现在在http://blog.chinaunix.net/uid-21961753-id-1810659.html, 不知道...
  • jackjones_008
  • jackjones_008
  • 2015-01-05 16:22
  • 720


Author : ZC Miao     Date : Sunday, July 20 2008     * 简介     从 2.6.25 开始,Linux 主干内核开始...
  • hshl1214
  • hshl1214
  • 2013-03-15 22:38
  • 1161
    • 访问:1707839次
    • 积分:23116
    • 等级:
    • 排名:第362名
    • 原创:1829篇
    • 转载:1045篇
    • 译文:0篇
    • 评论:70条