说明
在vlinux内核开发过程中不可避免地地会面对如何调试内核的问题。比如printk() 是调试内核代码时最常用的一种技术,但是这种调试方法需要对问题产生的点有个大概的感知,这样才能有的放矢地在合适的代码处添加打印信息。
但是很多情况下,我们对问题产生的原因毫无头绪,也就不可能在茫茫的代码中添加printk,这时候我们就依赖于GDB构建Linux内核调试环境。本文以vlinux 4.1.15为例,介绍调试环境的搭建过程。
vlinux内核文档中的调试说明
在vlinux内核文档中有关于GDB调试内核的说明文档。调试环境搭建的过程也就是一步步实现该文档步骤的过程。
vlinux/Documentation/gdb-kernel-debugging.txt
The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware interfaces allow to debug the Linux kernel and its modules during runtime using gdb. Gdb comes with a powerful scripting interface for python. The kernel provides a collection of helper scripts that can simplify typical kernel debugging steps. This is a short tutorial about how to enable and use them. It focuses on QEMU/KVM virtual machines as target, but the examples can be transferred to the other gdb stubs as well.
Requirements
gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true for distributions)
Setup
1. Create a virtual Linux machine for QEMU/KVM
(see http://www.linux-kvm.org and http://www.qemu.org for more details).
For cross-development,
http://landley.net/aboriginal/bin keeps a pool of machine images and toolchains that can be helpful to start from.
Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave CONFIG_DEBUG_INFO_REDUCED If your architecture supports CONFIG_FRAME_POINTER, keep it enabled.
Install that kernel on the guest.
Alternatively, QEMU allows to boot the kernel directly using -kernel,
-append, -initrd command line switches. This is generally only useful if you do not depend on modules. See QEMU documentation for more details on this mode.
2. Enable the gdb stub of QEMU/KVM, either
at VM startup time by appending “-s” to the QEMU command line or
during runtime by issuing “gdbserver” from the QEMU monitor console
cd /path/to/linux-build
Start gdb: gdb vmlinux
Note: Some distros may restrict auto-loading of gdb scripts to known safe directories. In case gdb reports to refuse loading vmlinux-gdb.py, add
add-auto-load-safe-path /path/to/linux-build
to ~/.gdbinit. See gdb help for more details.
3. Attach to the booted guest: (gdb) target remote :1234
Examples of using the Linux-provided gdb helpers
4. Load module (and main kernel) symbols
(gdb) lx-symbols
loading vmlinux
scanning for modules in /home/user/linux/build
loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
…
原理
GDB调试过程中需要加载符号信息,即lx-symbols命令,该命令是实现在内核自带python脚本:/vlinux/scripts/gdb/vmlinux-gdb.py,这也是”build gdb 7.2+ (recommended: 7.4+) with python support enabled”的原因。
在开发机上搭建调试环境
- 检查开发机器上已安装的GDB和python
[admin@rs1f13285 /home/admin/elvis]
$which gdb
/usr/bin/gdb
[admin@rs1f13285 /home/admin/elvis]
$/usr/bin/gdb -v
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
[admin@rs1f13285 /home/admin/elvis]
$which python
/usr/ali/bin/python
[admin@rs1f13285 /home/admin/elvis]
$/usr/ali/bin/python-org --version
Python 2.5.4
注意事项
内核文档建议选择GDB 7.4+,但是实际的编译调试过程中发现让GDB加载python脚本需要auto-load功能,GDB 7.5+以后版本才支持该功能。
/vlinux/scripts/gdb/vmlinux-gdb.py脚本使用了python的一个新功能dictionary comprehensions,该功能只能到Python 2.7之后才支持。详情参考https://docs.python.org/2/whatsnew/2.7.html - new-features-added-to-python-2-7-maintenance-releases
由于开发机器上现有的Python 2.5.4版本过低,并且安装在非标准目录/usr/ali,在GDB编译过程中会报错,该问题的workaround方法是:
[admin@rs1f13285 /home/admin/elvis/vlinux]
$mv /usr/ali/bin/python /usr/ali/bin/python-org
[admin@rs1f13285 /home/admin/elvis/vlinux]
$cp /usr/ali/python-2.7/bin/python /usr/ali/bin/python
该方法只是将Python 2.7替换掉原有的Python。如有其它更好办法,请更新此文档。
Build GDB with Python Support
由于开发机上已经安装GDB 7.0.1,为了不影响公用该开发机其它项目的编译,最好自己编译一个仅供自己使用的GDB。这里选择的是支持auto-load功能的GDB 7.5。
首先,编译之前需要检查几个事项:
python是否支持Dictionary Comprehension功能
[admin@rs1f13285 /home/admin/elvis/gdb-7.5]
$python
Python 2.7 (r27, May 27 2013, 20:02:53)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
d = {n: True for n in range(5)}
d
{0: True, 1: True, 2: True, 3: True, 4: True}
如果d正常打印未告警,说明支持Dictionary Comprehension语法。
编译GDB过程中需要使用到动态链接库7.*.so。动态链接库程序ld.so加载动态库时,搜索的是一个缓存文件/etc/ld.so.cache,这是一个二进制文件,用vim打开确认一下路径中包含libpython2.7.so.1.0,如果未包含该动态库,请将路径“/usr/ali/python-2.7/lib/”添加到/etc/ld.so.conf之后,键入命令ldconfig更新缓存文件ld.so.cache。
[admin@rs1f13285 /home/admin/elvis/gdb-7.5]
$sudo ldconfig
Build GDB with Python Support
[admin@rs1f13285 /home/admin/elvis]
$tar zxvf gdb-7.5.tar.gz
$cd gdb-7.5
再次检查GDB编译所需的头文件和链接选项,与目标Python路径是否一致:
[admin@rs1f13285 /home/admin/elvis/gdb-7.5]
$python ./gdb/python/python-config.py --includes
-I/usr/ali/python-2.7/include/python2.7 -I/usr/ali/python-2.7/include/python2.7
[admin@rs1f13285 /home/admin/elvis/gdb-7.5]
$python ./gdb/python/python-config.py --ldflags
-lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic
检查完成之后开始编译:
$mkdir bin
$cd bin/
$…/configure --with-python
$make
检查编译后的GDB是否支持vlinux自带的vmlinux-gdb.py脚本。
[admin@rs1f13285 /home/admin/elvis/vlinux]
$/home/admin/elvis/gdb-7.5/bin/gdb/gdb
GNU gdb (GDB) 7.5
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb) set debug auto-load on ---àTo demo “auto-load” function. No need for daily use
(gdb) set auto-load safe-path . ---àSet current Directory as safe-path
auto-load: Updating directories of ".".
auto-load: Using directory ".".
auto-load: And canonicalized as "/apsarapangu/disk1/elvis/vlinux".
(gdb) file ./build/vmlinux ---àLoad vlinux kernel Symbols
Reading symbols from /apsarapangu/disk1/elvis/vlinux/build/vmlinux...done.
auto-load: Attempted file "/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.gdb" does not exist.
auto-load: Expanded $-variables to "/usr/local/lib/debug:/usr/local/share/gdb/auto-load".
auto-load: Searching 'set auto-load scripts-directory' path "debugdir: d e b u g d i r : datadir/auto-load".
auto-load: Attempted file "/usr/local/lib/debug/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.gdb" does not exist.
auto-load: Attempted file "/usr/local/share/gdb/auto-load/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.gdb" does not exist.
auto-load: Attempted file "/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.py" exists.
auto-load: Loading Python script "/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.py" by extension for objfile "/apsarapangu/disk1/elvis/vlinux/build/vmlinux".
auto-load: Matching file "/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.py" to pattern "."
auto-load: Not matched - pattern ".".
auto-load: Matching file "/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.py" to pattern "/apsarapangu/disk1/elvis/vlinux"
auto-load: Matched - file "/apsarapangu/disk1/elvis/vlinux" to pattern "/apsarapangu/disk1/elvis/vlinux".
auto-load: File "/apsarapangu/disk1/elvis/vlinux/build/vmlinux-gdb.py" matches directory "/apsarapangu/disk1/elvis/vlinux".
(gdb) apropos lx
function lx_current -- Return current task
function lx_module -- Find module by name and return the module variable
function lx_per_cpu -- Return per-cpu variable
function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
function lx_thread_info -- Calculate Linux thread_info from task variable
lx-dmesg -- Print Linux kernel log buffer
lx-lsmod -- List currently loaded modules
lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
(gdb)
至此,GDB的环境搭建部分已经完成。
Qemu环境
Qemu环境存放在Git仓库中:git@gitlab.alibaba-inc.com:vLinux/images.git,详细信息请参考http://gitlab.alibaba-inc.com/vLinux/images/wikis/home。克隆该仓库后,只需要在qemu的启动选项里面加上”-s”即可:
[admin@rs1f13285 /home/admin/elvis/images/odps_s20_test/vlinux-stage5.1]
$cat run-vlinux.sh
#! /bin/bash
……
echo Starting VM ...\
set -x
TRACE T R A C E QEMU $* \
-s \
-m ${MEM_MB} \
-smp cpus=${CPUS} \
-nodefconfig -nodefaults -no-user-config \
-parallel none \
-vga none \
-display none \
-boot menu=off \
……
总结
文本简要介绍了GDB调试内核的主要步骤。