一、背景
qemu单步调试arm64 linux kernel-CSDN博客介绍了如何单步调试kernel, 但是我们经常写一些测试driver, driver的部分如何调试?
二、环境准备
调试driver 就需要准备一个简单的driver, 这里用最简单的hello world来演示如何调试,程序非常简单,生成一个字符设备,并且在cat的时候打印变量,还加了一个全局变量,用来gdb调试查看变量使用
#include <linux/module.h> /* Needed by all modules */
#include <linux/kernel.h> /* Needed for KERN_INFO */
#include <linux/init.h> /* Needed for the macros */
#include <linux/miscdevice.h> /* Needed for misc device */
int global_hello_value = 996;
#define DEVICE_NAME "Helloworld"
void test_for_debug(void)
{
printk(KERN_INFO "just for debug the text section %d", global_hello_value);
}
static int hello_open(struct inode *inode, struct file *file){
test_for_debug();
return 0;
}
static struct file_operations hello_fops = {
.owner = THIS_MODULE,
.open = hello_open,
};
static struct miscdevice hello_misc = {
.minor = MISC_DYNAMIC_MINOR,
.name = "Helloworld",
.fops = &hello_fops,
};
static int __init hello_start(void)
{
int ret;
ret = misc_register(&hello_misc);
if (ret < 0) {
printk(KERN_EMERG " %s register failed %d.\n", DEVICE_NAME, ret);
return ret;
}
printk(KERN_INFO "Hello world\n");
return 0;
}
static void __exit hello_end(void)
{
test_for_debug();
misc_deregister(&hello_misc);
printk(KERN_INFO "hello_end Goodbye\n");
}
MODULE_LICENSE("GPL");
MODULE_AUTHOR("geek");
MODULE_DESCRIPTION("A simple Hello world misc driver!");
MODULE_VERSION("0.1");
module_init(hello_start);
module_exit(hello_end);
编译ko需要使用 make modules指令
make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- modules -j8
编译完成后,会在hello driver的源码目录生成对应的ko文件;
将ko 放在我们的rootfs 中,参考前一篇 无人知晓:qemu搭建arm64 linux kernel调试环境 rootfs image制作部分,挂载rootfs.img, 拷贝ko, 然后umount img
sudo mount rootfs.img rootfs
sudo mkdir rootfs/driver
sudo cp ../hello_driver.ko rootfs/driver/
sudo umount rootfs
三、加载ko
insmod 加载打印hello world, 同时生成/dev/Helloworld节点
~ # insmod driver/hello_driver.ko
[ 11.203785] Hello world
~ # cd /dev/
/dev # ls
Helloworld ptypc tty36 ttyp0
gdb 连接后,尝试看下之前ko的global_hello_value变量,无法显示,test_for_debug函数也是无法找到的
(gdb) p global_hello_value
No symbol "global_hello_value" in current context.
(gdb) b test_for_debug
Function "test_for_debug" not defined.
Make breakpoint pending on future shared library load? (y or [n]) n
四、调试ko
调试ko的步骤也是分三步:先加载ko, 确认ko加载的地址;然后加载ko symbols,加载地址需要和实际加载一致 ; 最后一步就是加断点,然后启动调试即可
方法1: insmod ko之后根据节点/sys/module/XXXX/sections/ 确认ko 加载的地址
/driver # cat /sys/module/hello_driver/sections/.text
0xffff80007a860000
加载ko symbols
gdb 中执行
(gdb) add-symbol-file drivers/my_driver/hello_driver.ko -s .text 0xffff80007a860000
add symbol table from file "drivers/my_driver/hello_driver.ko" at
.text_addr = 0xffff80007a860000
(y or n) y
Reading symbols from drivers/my_driver/hello_driver.ko...
然后再设置断点
(gdb) b test_for_debug
Breakpoint 1 at 0x4: test_for_debug. (3 locations)
(gdb) c
Continuing.
continue 后 qemu中cat节点即可触发中断
/dev # cat Helloworld
gdb窗口
(gdb) bt
#0 0xffff80007a860004 in test_for_debug () at drivers/my_driver/hello_driver.c:12
#1 hello_open (inode=0xffff000002f11610, file=0xffff00000381bb00) at drivers/my_driver/hello_driver.c:16
#2 0xffff8000808623d8 in misc_open (inode=0xffff000002f11610, file=0xffff00000381bb00) at drivers/char/misc.c:165
#3 0xffff8000802c8250 in chrdev_open (inode=0xffff000002f11610, filp=0xffff00000381bb00) at fs/char_dev.c:414
#4 0xffff8000802bcfcc in do_dentry_open (f=0xffff00000381bb00, inode=0xffff000002f11610, open=0xffff8000802c8194 <chrdev_open>) at fs/open.c:929
#5 0xffff8000802beda0 in vfs_open (path=0x2 <hello_end+2>, file=0xffff000002f11610) at fs/open.c:1063
#6 0xffff8000802d6350 in do_open (op=<optimized out>, file=<optimized out>, nd=<optimized out>) at fs/namei.c:3640
#7 path_openat (nd=0xffff800082adbc40, op=0x4 <hello_end+4>, flags=2055602176) at fs/namei.c:3797
#8 0xffff8000802d706c in do_filp_open (dfd=-100, pathname=0xffff000002e40000, op=0xffff800082adbd74) at fs/namei.c:3824
#9 0xffff8000802bf048 in do_sys_openat2 (dfd=-100, filename=0x646c <error: Cannot access memory at address 0x646c>, how=0x3 <hello_end+3>) at fs/open.c:1422
#10 0xffff8000802bf388 in do_sys_open (mode=<optimized out>, flags=<optimized out>, filename=<optimized out>, dfd=<optimized out>) at fs/open.c:1437
#11 __do_sys_openat (mode=<optimized out>, flags=<optimized out>, filename=<optimized out>, dfd=<optimized out>) at fs/open.c:1453
#12 __se_sys_openat (mode=<optimized out>, flags=<optimized out>, filename=<optimized out>, dfd=<optimized out>) at fs/open.c:1448
#13 __arm64_sys_openat (regs=0xffff80007a860000 <hello_open>) at fs/open.c:1448
#14 0xffff800080027738 in __invoke_syscall (syscall_fn=<optimized out>, regs=<optimized out>) at arch/arm64/kernel/syscall.c:37
#15 invoke_syscall (regs=0xffff800082adbeb0, scno=58833664, sc_nr=2055602176, syscall_table=0x2 <hello_end+2>) at arch/arm64/kernel/syscall.c:51
#16 0xffff800080027840 in el0_svc_common (regs=0xffff800082adbeb0, scno=1, syscall_table=0x2 <hello_end+2>, sc_nr=<optimized out>) at arch/arm64/kernel/syscall.c:136
#17 0xffff8000800278fc in do_el0_svc (regs=0xffff000002f11610) at arch/arm64/kernel/syscall.c:155
#18 0xffff800081016224 in el0_svc (regs=0xffff800082adbeb0) at arch/arm64/kernel/entry-common.c:678
#19 0xffff800081016688 in el0t_64_sync_handler (regs=0xffff80007a860000 <hello_open>) at arch/arm64/kernel/entry-common.c:696
#20 0xffff800080011d4c in el0t_64_sync () at arch/arm64/kernel/entry.S:59
但是还有一个问题,查看global_hello_value怎么失败了?
(gdb) p global_hello_value
Cannot access memory at address 0x288
这是因为,前面我加载symbols时没有指定.data段,只指定了.text段
退出gdb,重新修改ko 加载symbols 指令为(.text, .data 等段信息均在/sys/module/hello_driver/sections/ 下)
(gdb) add-symbol-file drivers/my_driver/hello_driver.ko -s .text 0xffff80007a860000 -s .data 0xffff80007a862000
add symbol table from file "drivers/my_driver/hello_driver.ko" at
.text_addr = 0xffff80007a860000
.data_addr = 0xffff80007a862000
(y or n) y
Reading symbols from drivers/my_driver/hello_driver.ko...
(gdb) p global_hello_value
$1 = 996
方法2:增加断点,在load_module中停住,这样可以debug 初始化的部分,比如module_init中的函数(如果我们驱动在这里有bug,根本没机会生成/sys/module下的driver 段信息)
ko加载的可以参考这篇 linux ko模块动态加载源码分析
核心就是在do_init_module 设置断点,然后从结构体struct module 提取module加载信息
注意:module_init函数在 .init.text 段,add-symbol-file 需要加入这个 .init.text ,才能在module init设置断点
(gdb)b do_init_module
设置好断点后继续,然后加载ko,触发断点后,将struct module的段信息及地址信息用gdb 显示出来
(gdb) n
2523 freeinit->init_text = mod->mem[MOD_INIT_TEXT].base;
(gdb) p *mod->sect_attrs->attrs@20
在hello_start 设置断点
(gdb) add-symbol-file drivers/my_driver/hello_driver.ko -s .text 18446603338276798464 -s .init.text 18446603338276823040 -s .data 18446603338276806656
add symbol table from file "drivers/my_driver/hello_driver.ko" at
.text_addr = 0xffff80007a860000
.init.text_addr = 0xffff80007a866000
.data_addr = 0xffff80007a862000
(y or n) y
Reading symbols from drivers/my_driver/hello_driver.ko...
(gdb) b hello_start
Breakpoint 2 at 0xffff80007a866000: file drivers/my_driver/hello_driver.c, line 34.
Thread 2 hit Breakpoint 2, hello_start () at drivers/my_driver/hello_driver.c:34
warning: Source file is more recent than executable.
34 ret = misc_register(&hello_misc);
(gdb) bt
#0 hello_start () at drivers/my_driver/hello_driver.c:34
#1 0xffff800080014dbc in do_one_initcall (fn=0xffff80007a866000 <hello_start>) at init/main.c:1232
#2 0xffff800080120d20 in do_init_module (mod=0xffff80007a862180) at kernel/module/main.c:2530
#3 0xffff800080122dfc in load_module (info=0xffff800082af3ac8, uargs=0xffff0000035e9d80 "\004", flags=0) at kernel/module/main.c:2981
#4 0xffff800080123020 in __do_sys_init_module (umod=0x23756d60, len=38752, uargs=0x5bdf21 "") at kernel/module/main.c:3058
#5 0xffff800080123140 in __se_sys_init_module (uargs=<optimized out>, len=<optimized out>, umod=<optimized out>) at kernel/module/main.c:3038
#6 __arm64_sys_init_module (regs=0x0 <hello_end>) at kernel/module/main.c:3038
#7 0xffff800080027738 in __invoke_syscall (syscall_fn=<optimized out>, regs=<optimized out>) at arch/arm64/kernel/syscall.c:37
#8 invoke_syscall (regs=0xffff800082af3eb0, scno=56532352, sc_nr=0, syscall_table=0x0 <hello_end>) at arch/arm64/kernel/syscall.c:51
#9 0xffff800080027840 in el0_svc_common (regs=0xffff800082af3eb0, scno=-48, syscall_table=0x0 <hello_end>, sc_nr=<optimized out>) at arch/arm64/kernel/syscall.c:136
#10 0xffff8000800278fc in do_el0_svc (regs=0x0 <hello_end>) at arch/arm64/kernel/syscall.c:155
#11 0xffff800081016224 in el0_svc (regs=0xffff800082af3eb0) at arch/arm64/kernel/entry-common.c:678
#12 0xffff800081016688 in el0t_64_sync_handler (regs=0x0 <hello_end>) at arch/arm64/kernel/entry-common.c:696
#13 0xffff800080011d4c in el0t_64_sync () at arch/arm64/kernel/entry.S:595
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
方法3:利用vmlinux-gdb.py脚本中的lx-symbols辅助命令
利用gdb脚本加载, 也是三步,先构建gdb script环境; 去掉gdb 脚本执行限制 ; 调用lx-symbols获取ko加载地址,当让script中除了获取lx-symbols外还有很多其他脚本辅助我们调试;
参考:https://www.codenong.com/cs105354913/
构建脚本环境
make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- scripts_gdb
根目录生成vmlinux-gdb.py表示成功
去掉gdb脚本加载限制,在~/.config/gdb/gdbinit 中添加set auto-load safe-path / 环境变量;
此时在linux source加载vmlinux也会提示这个限制错误及解决方法
添加环境变量后重新启动gdb,出现了加载vmlinux-gdb.py脚本的错误
上面的报错原因是因为编译kernel版本中打开了config CONFIG_DEBUG_INFO_REDUCED, 这个会影响gdb script的完整功能,在arch/arm64/ configs/defconfig中去掉这个config,重新编辑后加载vmlinux;
编译kernel之后记得还需要重新生成下脚本
make ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- scripts_gdb
(gdb) lx-symbols
loading vmlinux
scanning for modules in /home/geek/workspace/linux/linux-6.6.1
loading @0xffff80007a860000: /home/geek/workspace/linux/linux-6.6.1/drivers/my_driver/hello_driver.ko
这个指令会自动加载ko symbols, 直接加断点即可调试,免去了前面通过 add-symbol-file 设置ko section的繁琐步骤;(但是如果你是调试驱动 init的部分,还是得使用上面的方法2)
方法3虽然配置有些麻烦,但是vmlinux-gdb.py 提供了非常多的调试功能 apropos lx 可以查看,这些调试功能能帮助我们提升调试的效率
(gdb) apropos lx
function lx_clk_core_lookup -- Find struct clk_core by name
function lx_current -- Return current task.
function lx_dentry_name -- Return string of the full path of a dentry.
function lx_device_find_by_bus_name -- Find struct device by bus and name (both strings)
function lx_device_find_by_class_name -- Find struct device by class and name (both strings)
function lx_i_dentry -- Return dentry pointer for inode.
function lx_module -- Find module by name and return the module variable.
function lx_per_cpu -- Return per-cpu variable.
function lx_radix_tree_lookup -- Lookup and return a node from a RadixTree.
function lx_rb_first -- Lookup and return a node from an RBTree
function lx_rb_last -- Lookup and return a node from an RBTree.
function lx_rb_next -- Lookup and return a node from an RBTree.
function lx_rb_prev -- Lookup and return a node from an RBTree.
function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable.
function lx_thread_info -- Calculate Linux thread_info from task variable.
function lx_thread_info_by_pid -- Calculate Linux thread_info from task variable found by pid
lx-clk-summary -- Print clk tree summary
lx-cmdline -- Report the Linux Commandline used in the current kernel.
lx-configdump -- Output kernel config to the filename specified as the command
lx-cpus -- List CPU status arrays
lx-device-list-bus -- Print devices on a bus (or all buses if not specified)
lx-device-list-class -- Print devices in a class (or all classes if not specified)
lx-device-list-tree -- Print a device and its children recursively
lx-dmesg -- Print Linux kernel log buffer.
lx-dump-page-owner -- Dump page owner
lx-fdtdump -- Output Flattened Device Tree header and dump FDT blob to the filename
lx-genpd-summary -- Print genpd summary
lx-getmod-by-textaddr -- Look up loaded kernel module by text address.
lx-interruptlist -- Print /proc/interrupts
lx-iomem -- Identify the IO memory resource locations defined by the kernel
lx-ioports -- Identify the IO port resource locations defined by the kernel
lx-list-check -- Verify a list consistency
lx-lsmod -- List currently loaded modules.
lx-mounts -- Report the VFS mounts of the current process namespace.
lx-page_address -- struct page to linear mapping address
lx-page_to_pfn -- struct page to PFN
lx-page_to_phys -- struct page to physical address
lx-pfn_to_kaddr -- PFN to kernel address
lx-pfn_to_page -- PFN to struct page
lx-ps -- Dump Linux tasks.
lx-slabinfo -- Show slabinfo
lx-slabtrace -- Show specific cache slabtrace
lx-sym_to_pfn -- symbol address to PFN
lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules.
lx-timerlist -- Print /proc/timer_list
lx-version -- Report the Linux Version of the current kernel.
lx-virt_to_page -- virtual address to struct page
lx-virt_to_phys -- virtual address to physical address
lx-vmallocinfo -- Show vmallocinfo
比如我想要查看所有task的信息,执行lx-ps即可,如果要看task_struct的详细信息,将显示的地址转换成struct task_struct *指针即可显示
比如查看一个进程的进程名
(gdb) p ((struct task_struct*)0xffff000002d68ec0)->comm
$6 = "watchdogd\000\000\000\000\000\000"