LDD3学习6--Linux内核的调试-CSDN博客

本文链接：https://blog.csdn.net/fanged/article/details/142071705

注：对应LDD3第四章

1 内核配置

在编译内核时候，有一个kernel hacking菜单，可以配置内核的一些调试选项。

2 常规方法

用的最多的还是printk了。

其实我一直想的是printk可以直接输出到屏幕，不用dmesg。所以专门写一下。

试验一下几个参数：

    printk(KERN_EMERG "This is an informational message1.\n");
    printk(KERN_ALERT "This is an informational message2.\n");
    printk(KERN_CRIT "This is an informational message3.\n");
    printk(KERN_ERR "This is an informational message4.\n");
    printk(KERN_WARNING "This is an informational message5.\n");
    printk(KERN_NOTICE "This is an informational message6.\n");
    printk(KERN_INFO "This is an informational message7.\n");
    printk(KERN_DEBUG "This is an informational message8.\n");
    pr_err("debug_init\n");

这里要注意，KERN_EMERG后面是没有逗号的。

结果如下：

[ 3203.071708] This is an informational message1.
[ 3203.076223] This is an informational message2.
[ 3203.080719] This is an informational message3.

看来以后要即时输出，可以用KERN_EMERG，KERN_ALERT，KERN_CRIT这几个。

看内核哪些可以打印，用这个：

cat /proc/sys/kernel/printk

第一个就是打印级别，比它小的都能打印。

对了，pr_err这几个实际是printk的封装。和上面printk的参数是对应的。详细的如下：

pr_emerg() - 紧急情况，系统可能处于不可用的状态。
pr_alert() - 需要立即注意的情况。
pr_crit() - 临界情况，表示严重问题。
pr_err() - 错误情况，通常用于报告可恢复的错误。
pr_warn() - 警告信息，用于提醒可能的问题。
pr_notice() - 重要的提示信息，比警告级别低。
pr_info() - 一般信息，用于常规信息的记录。
pr_debug() - 调试信息，通常用于开发和测试阶段。

还是简单看看linux内核的log系统

Linux 内核的 klog 系统是指与内核日志相关的一系列工具和守护进程，它们负责捕获、处理和发送内核生成的日志消息。这些工具主要包括 klogd 和 dmesg。

klogd：这是一个守护进程，专门用于截获并记录 Linux 内核消息。它可以将日志消息输出到控制台、文件或 syslogd 守护进程等。klogd 通过读取 /proc/kmsg 文件来获取内核日志，这个文件是内核日志消息的环形缓冲区。klogd 可以将日志消息传递给 syslogd，由 syslogd 统一处理并记录到文件中，如 /var/log/messages 或 /var/log/syslog。
dmesg：这是一个命令行工具，用于显示和控制内核环形缓冲区中的日志消息。dmesg 可以显示内核启动时的日志消息，也可以用于实时查看内核日志。它提供了多种选项来过滤和显示日志消息，例如 -c 清除日志缓冲区，-s 设置缓冲区大小，-n 设置日志级别等。
syslogd：这是一个系统日志守护进程，负责记录系统中的日志信息，包括内核日志和应用程序日志。syslogd 根据配置文件 /etc/syslog.conf 中的规则来决定如何处理和存储日志消息。
rsyslog：在现代 Linux 系统中，rsyslog 通常取代了 syslogd，提供了更强大的日志处理功能，包括日志过滤、格式化和转发等。
/proc/kmsg：这是一个特殊的文件，提供了从内核日志缓冲区读取日志消息的接口。klogd 和 dmesg 都可以通过读取 /proc/kmsg 来获取日志消息。
/var/log：这是系统日志文件的存放目录，包含了各种系统和应用程序的日志文件，如 /var/log/messages、/var/log/syslog 等。

也就是说printk和printf是不同的，printf是直接往stdout写，printk是写到内核日志消息的环形缓冲区里面。这个由klogd管理，可以用dmesg查看。（感觉这部分和Android的logcat很像。。）

再看看别的几个调试宏。

BUG()

打印堆栈信息，然后整个程序退出。

pi@raspberrypi:~/debug$ sudo rmmod debug.ko;sudo insmod debug.ko
[  659.200377] ------------[ cut here ]------------
[  659.205022] kernel BUG at /home/pi/debug/debug.c:11!
[  659.209993] Internal error: Oops - BUG: 0 [#1] SMP ARM
[  659.215160] Modules linked in: debug(O+) brcmfmac_wcc vc4 binfmt_misc brcmfmac snd_soc_hdmi_codec drm_display_helper cec brcmutil drm_dma_helper drm_kms_helper cfg80211 snd_soc_core bcm2835_codec(C) bcm2835_v4l2(C) v4l2_mem2mem bcm2835_isp(C) snd_compress rfkill bcm2835_mmal_vchiq(C) snd_pcm_dmaengine videobuf2_dma_contig videobuf2_vmalloc videobuf2_memops raspberrypi_hwmon videobuf2_v4l2 videodev snd_bcm2835(C) snd_pcm videobuf2_common snd_timer mc snd vc_sm_cma(C) raspberrypi_gpiomem uio_pdrv_genirq uio drm fuse dm_mod drm_panel_orientation_quirks backlight ip_tables x_tables ipv6 spidev i2c_bcm2835 spi_bcm2835 fixed [last unloaded: debug(O)]
[  659.272473] CPU: 0 PID: 1714 Comm: insmod Tainted: G        WC O       6.6.31+rpt-rpi-v7 #1  Raspbian 1:6.6.31-1+rpt1
[  659.283095] Hardware name: BCM2835
[  659.286495] PC is at debug_init+0x8/0x1000 [debug]
[  659.291304] LR is at do_one_initcall+0x50/0x214
[  659.295843] pc : [<7f472008>]    lr : [<80101f84>]    psr: 60000013
[  659.302112] sp : bbbe9e58  ip : bbbe9e44  fp : 823ed540
[  659.307337] r10: aeb03030  r9 : 823ed540  r8 : 00000000
[  659.312563] r7 : 81fe8000  r6 : 0002ed38  r5 : 8264c200  r4 : 7f472000
[  659.319093] r3 : 81fe8000  r2 : 00000000  r1 : 00000000  r0 : 00000000
[  659.325622] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  659.332762] Control: 10c5383d  Table: 03c1c06a  DAC: 00000055
[  659.338507] Register r0 information: NULL pointer
[  659.343216] Register r1 information: NULL pointer
[  659.347922] Register r2 information: NULL pointer
[  659.352628] Register r3 information: slab task_struct start 81fe8000 pointer offset 0 size 4544
[  659.361348] Register r4 information: 1-page vmalloc region starting at 0x7f472000 allocated at load_module+0x6ec/0x1c58
[  659.372148] Register r5 information: slab kmalloc-64 start 8264c200 pointer offset 0 size 64
[  659.380605] Register r6 information: non-paged memory
[  659.385659] Register r7 information: slab task_struct start 81fe8000 pointer offset 0 size 4544
[  659.394376] Register r8 information: NULL pointer
[  659.399082] Register r9 information: slab kmalloc-cg-192 start 823ed540 pointer offset 0 size 192
[  659.407973] Register r10 information: slab ext4_inode_cache start aeb02f58 pointer offset 216 size 800
[  659.417299] Register r11 information: slab kmalloc-cg-192 start 823ed540 pointer offset 0 size 192
[  659.426277] Register r12 information: 2-page vmalloc region starting at 0xbbbe8000 allocated at kernel_clone+0xac/0x3a8
[  659.437076] Process insmod (pid: 1714, stack limit = 0xbd08114b)
[  659.443087] Stack: (0xbbbe9e58 to 0xbbbea000)
[  659.447446] 9e40:                                                       a0000013 ffffffff
[  659.455631] 9e60: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  659.463816] 9e80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  659.472001] 9ea0: 00000000 5e892de7 7f34f040 8264c200 0002ed38 00000000 81296740 801ad5b8
[  659.480186] 9ec0: 00000000 81296740 00000000 823ed540 0002ed38 801af768 bbbe9eec 7fffffff
[  659.488371] 9ee0: 00000000 00000002 0001fca8 bb8b7000 bb8b74b6 bb8b7100 bb8b7000 0000141c
[  659.496556] 9f00: bb8b7e2c bb8b7cb4 bb8b7a94 00000340 00000450 000005cc 000004d7 00000000
[  659.504740] 9f20: 000005bc 00000023 00000024 0000000f 0000001d 0000001a 00000000 5e892de7
[  659.512925] 9f40: 00000081 81296330 00000001 801afab8 00000017 0002ed38 00000000 aeb03030
[  659.521110] 9f60: 00000000 81296544 00000000 00000000 bbbe9f70 bbbe9f70 00000000 5e892de7
[  659.529295] 9f80: 00000002 00000002 7ecfddb4 a65cfd00 0000017b 80100298 81fe8000 0000017b
[  659.537479] 9fa0: 0004fbec 80100040 00000002 7ecfddb4 00000003 0002ed38 00000000 7ecfdfe6
[  659.545664] 9fc0: 00000002 7ecfddb4 a65cfd00 0000017b 00b1ecf8 00000000 7ecfded4 0004fbec
[  659.553849] 9fe0: 7ecfdbc8 7ecfdbb8 00024640 76b2bed0 60000010 00000003 00000000 00000000
[  659.562051]  debug_init [debug] from do_one_initcall+0x50/0x214
[  659.567990]  do_one_initcall from do_init_module+0x5c/0x20c
[  659.573573]  do_init_module from init_module_from_file+0x9c/0xd8
[  659.579588]  init_module_from_file from sys_finit_module+0x154/0x290
[  659.585951]  sys_finit_module from ret_fast_syscall+0x0/0x4c
[  659.591618] Exception stack(0xbbbe9fa8 to 0xbbbe9ff0)
[  659.596672] 9fa0:                   00000002 7ecfddb4 00000003 0002ed38 00000000 7ecfdfe6
[  659.604857] 9fc0: 00000002 7ecfddb4 a65cfd00 0000017b 00b1ecf8 00000000 7ecfded4 0004fbec
[  659.613040] 9fe0: 7ecfdbc8 7ecfdbb8 00024640 76b2bed0
[  659.618095] Code: bad PC value
[  659.621151] ---[ end trace 0000000000000000 ]---
[  659.625768] note: insmod[1714] exited with irqs disabled
Segmentation fault

panic("BUG!");

这玩意威力是真的大，直接整个系统崩了。

root@raspberrypi:/home/pi# [ 4269.123720] Kernel panic - not syncing: BUG!
[ 4269.128020] CPU: 0 PID: 1498 Comm: insmod Tainted: G         C O       6.6.31+rpt-rpi-v7 #1  Raspbian 1:6.6.31-1+rpt1
[ 4269.138641] Hardware name: BCM2835
[ 4269.142048]  unwind_backtrace from show_stack+0x18/0x1c
[ 4269.147292]  show_stack from dump_stack_lvl+0x50/0x68
[ 4269.152354]  dump_stack_lvl from panic+0x114/0x33c
[ 4269.157153]  panic from debug_init+0x18/0x1000 [debug]
[ 4269.162324] unwind: Index not found 7f488018
[ 4269.166598] CPU2: stopping
[ 4269.166602] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G         C O       6.6.31+rpt-rpi-v7 #1  Raspbian 1:6.6.31-1+rpt1
[ 4269.166609] Hardware name: BCM2835
[ 4269.166613]  unwind_backtrace from show_stack+0x18/0x1c
[ 4269.166626]  show_stack from dump_stack_lvl+0x50/0x68
[ 4269.166636]  dump_stack_lvl from do_handle_IPI+0x18c/0x1c0
[ 4269.166647]  do_handle_IPI from ipi_handler+0x20/0x28
[ 4269.166658]  ipi_handler from handle_percpu_devid_irq+0x84/0x1c0
[ 4269.166670]  handle_percpu_devid_irq from generic_handle_domain_irq+0x30/0x40
[ 4269.166683]  generic_handle_domain_irq from bcm2836_arm_irqchip_handle_ipi+0x94/0xb4
[ 4269.166698]  bcm2836_arm_irqchip_handle_ipi from generic_handle_domain_irq+0x30/0x40
[ 4269.166709]  generic_handle_domain_irq from generic_handle_arch_irq+0x34/0x44
[ 4269.166721]  generic_handle_arch_irq from call_with_stack+0x18/0x20
[ 4269.166737]  call_with_stack from __irq_svc+0x84/0x94
[ 4269.166751] Exception stack(0xbb875f70 to 0xbb875fb8)
[ 4269.166757] 5f60:                                     00000003 00000001 00000002 60000093
[ 4269.166764] 5f80: 815ac700 81105018 815ac700 81105074 0000406a 410fd034 00000000 00000000
[ 4269.166770] 5fa0: 81103d80 bb875fc0 80b3106c 80b314c4 60000013 ffffffff
[ 4269.166774]  __irq_svc from default_idle_call+0x38/0xa0
[ 4269.166784]  default_idle_call from do_idle+0xac/0x118
[ 4269.166798]  do_idle from cpu_startup_entry+0x30/0x34
[ 4269.166812]  cpu_startup_entry from secondary_start_kernel+0x11c/0x124
[ 4269.166825]  secondary_start_kernel from 0x1015a0
[ 4269.166837] CPU3: stopping
[ 4269.166841] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G         C O       6.6.31+rpt-rpi-v7 #1  Raspbian 1:6.6.31-1+rpt1
[ 4269.166848] Hardware name: BCM2835
[ 4269.166852]  unwind_backtrace from show_stack+0x18/0x1c
[ 4269.166866]  show_stack from dump_stack_lvl+0x50/0x68
[ 4269.166875]  dump_stack_lvl from do_handle_IPI+0x18c/0x1c0
[ 4269.166885]  do_handle_IPI from ipi_handler+0x20/0x28
[ 4269.166896]  ipi_handler from handle_percpu_devid_irq+0x84/0x1c0
[ 4269.166907]  handle_percpu_devid_irq from generic_handle_domain_irq+0x30/0x40
[ 4269.166918]  generic_handle_domain_irq from bcm2836_arm_irqchip_handle_ipi+0x94/0xb4
[ 4269.166930]  bcm2836_arm_irqchip_handle_ipi from generic_handle_domain_irq+0x30/0x40
[ 4269.166942]  generic_handle_domain_irq from generic_handle_arch_irq+0x34/0x44
[ 4269.166954]  generic_handle_arch_irq from call_with_stack+0x18/0x20
[ 4269.166967]  call_with_stack from __irq_svc+0x84/0x94
[ 4269.166980] Exception stack(0xbb879f70 to 0xbb879fb8)
[ 4269.166986] 9f60:                                     00000003 00000001 00000003 60000093
[ 4269.166993] 9f80: 815ad8c0 81105018 815ad8c0 81105074 0000406a 410fd034 00000000 00000000
[ 4269.166999] 9fa0: 81103d80 bb879fc0 80b3106c 80b314c4 60000013 ffffffff
[ 4269.167002]  __irq_svc from default_idle_call+0x38/0xa0
[ 4269.167013]  default_idle_call from do_idle+0xac/0x118
[ 4269.167025]  do_idle from cpu_startup_entry+0x30/0x34
[ 4269.167038]  cpu_startup_entry from secondary_start_kernel+0x11c/0x124
[ 4269.167051]  secondary_start_kernel from 0x1015a0
[ 4269.167063] CPU1: stopping
[ 4269.167066] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G         C O       6.6.31+rpt-rpi-v7 #1  Raspbian 1:6.6.31-1+rpt1
[ 4269.167073] Hardware name: BCM2835
[ 4269.167077]  unwind_backtrace from show_stack+0x18/0x1c
[ 4269.167091]  show_stack from dump_stack_lvl+0x50/0x68
[ 4269.167100]  dump_stack_lvl from do_handle_IPI+0x18c/0x1c0
[ 4269.167111]  do_handle_IPI from ipi_handler+0x20/0x28
[ 4269.167122]  ipi_handler from handle_percpu_devid_irq+0x84/0x1c0
[ 4269.167132]  handle_percpu_devid_irq from generic_handle_domain_irq+0x30/0x40
[ 4269.167144]  generic_handle_domain_irq from bcm2836_arm_irqchip_handle_ipi+0x94/0xb4
[ 4269.167155]  bcm2836_arm_irqchip_handle_ipi from generic_handle_domain_irq+0x30/0x40
[ 4269.167167]  generic_handle_domain_irq from generic_handle_arch_irq+0x34/0x44
[ 4269.167179]  generic_handle_arch_irq from call_with_stack+0x18/0x20
[ 4269.167193]  call_with_stack from __irq_svc+0x84/0x94
[ 4269.167206] Exception stack(0xbb871f70 to 0xbb871fb8)
[ 4269.167212] 1f60:                                     00000003 00000001 00000001 60000093
[ 4269.167218] 1f80: 815ab540 81105018 815ab540 81105074 0000406a 410fd034 00000000 00000000
[ 4269.167224] 1fa0: 81103d80 bb871fc0 80b3106c 80b314c4 60000013 ffffffff
[ 4269.167228]  __irq_svc from default_idle_call+0x38/0xa0
[ 4269.167238]  default_idle_call from do_idle+0xac/0x118
[ 4269.167250]  do_idle from cpu_startup_entry+0x30/0x34
[ 4269.167264]  cpu_startup_entry from secondary_start_kernel+0x11c/0x124
[ 4269.167277]  secondary_start_kernel from 0x1015a0
[ 4269.584859] ---[ end Kernel panic - not syncing: BUG! ]---

WARN

WARN(1, "Some condition is true, and this is the warning message!");
这个会打印堆栈，不会OOPS。

[  289.274457] Some condition is true, and this is the warning message!
[  289.274463] Modules linked in: debug(O+) brcmfmac_wcc vc4 binfmt_misc brcmfmac snd_soc_hdmi_codec drm_display_helper cec brcmutil drm_dma_helper drm_kms_helper cfg80211 snd_soc_core bcm2835_codec(C) bcm2835_v4l2(C) v4l2_mem2mem bcm2835_isp(C) snd_compress rfkill bcm2835_mmal_vchiq(C) snd_pcm_dmaengine videobuf2_dma_contig videobuf2_vmalloc videobuf2_memops raspberrypi_hwmon videobuf2_v4l2 videodev snd_bcm2835(C) snd_pcm videobuf2_common snd_timer mc snd vc_sm_cma(C) raspberrypi_gpiomem uio_pdrv_genirq uio drm fuse dm_mod drm_panel_orientation_quirks backlight ip_tables x_tables ipv6 spidev i2c_bcm2835 spi_bcm2835 fixed
[  289.274794] CPU: 2 PID: 1441 Comm: insmod Tainted: G         C O       6.6.31+rpt-rpi-v7 #1  Raspbian 1:6.6.31-1+rpt1
[  289.274813] Hardware name: BCM2835
[  289.274832]  unwind_backtrace from show_stack+0x18/0x1c
[  289.274870]  show_stack from dump_stack_lvl+0x50/0x68
[  289.274898]  dump_stack_lvl from __warn+0x80/0x11c
[  289.274926]  __warn from warn_slowpath_fmt+0x12c/0x198
[  289.274953]  warn_slowpath_fmt from debug_init+0x28/0x1000 [debug]
[  289.275013]  debug_init [debug] from do_one_initcall+0x50/0x214
[  289.275055]  do_one_initcall from do_init_module+0x5c/0x20c
[  289.275087]  do_init_module from init_module_from_file+0x9c/0xd8
[  289.275113]  init_module_from_file from sys_finit_module+0x154/0x290
[  289.275140]  sys_finit_module from ret_fast_syscall+0x0/0x4c
[  289.275165] Exception stack(0xbbbc5fa8 to 0xbbbc5ff0)
[  289.275183] 5fa0:                   00000002 7edfadb4 00000003 0002ed38 00000000 7edfafe6
[  289.275202] 5fc0: 00000002 7edfadb4 da8b6300 0000017b 010e1cf8 00000000 7edfaed4 0004fbec
[  289.275217] 5fe0: 7edfabc8 7edfabb8 00024640 76b0bed0
[  289.275229] ---[ end trace 0000000000000000 ]---

dump_stack()
这个会打印堆栈，但是信息比上面的好像少一些。

[  430.477197] Hardware name: BCM2835
[  430.477213]  unwind_backtrace from show_stack+0x18/0x1c
[  430.477247]  show_stack from dump_stack_lvl+0x50/0x68
[  430.477269]  dump_stack_lvl from debug_init+0x10/0x1000 [debug]
[  430.477317]  debug_init [debug] from do_one_initcall+0x50/0x214
[  430.477350]  do_one_initcall from do_init_module+0x5c/0x20c
[  430.477374]  do_init_module from init_module_from_file+0x9c/0xd8
[  430.477394]  init_module_from_file from sys_finit_module+0x154/0x290
[  430.477414]  sys_finit_module from ret_fast_syscall+0x0/0x4c
[  430.477432] Exception stack(0xbc0b1fa8 to 0xbc0b1ff0)
[  430.477447] 1fa0:                   00000002 7ed67db4 00000003 0002ed38 00000000 7ed67fe6
[  430.477461] 1fc0: 00000002 7ed67db4 fa0eb200 0000017b 00176cf8 00000000 7ed67ed4 0004fbec
[  430.477472] 1fe0: 7ed67bc8 7ed67bb8 00024640 76ab3ed0
[  430.477481] debug_init

3 使用/proc调试

这个是驱动自己提供的一个接口。好处是数据需要用的时候才生成，而不是一直往一个地方写。

比如：

int scull_read_procmem(struct seq_file *s, void *v)
{
        int i, j;
        int limit = s->size - 80; /* Don't print more than this */

        for (i = 0; i < scull_nr_devs && s->count <= limit; i++) {
                struct scull_dev *d = &scull_devices[i];
                struct scull_qset *qs = d->data;
                if (mutex_lock_interruptible(&d->lock))
                        return -ERESTARTSYS;
                seq_printf(s,"\nDevice %i: qset %i, q %i, sz %li\n",
                             i, d->qset, d->quantum, d->size);
                for (; qs && s->count <= limit; qs = qs->next) { /* scan the list */
                        seq_printf(s, "  item at %p, qset at %p\n",
                                     qs, qs->data);
                        if (qs->data && !qs->next) /* dump only the last item */
                                for (j = 0; j < d->qset; j++) {
                                        if (qs->data[j])
                                                seq_printf(s, "    % 4i: %8p\n",
                                                             j, qs->data[j]);
                                }
                }
                mutex_unlock(&scull_devices[i].lock);
        }
        return 0;
}

4 使用 kgdb 进行内核调试

如果是用的gdb和qemu，可以使用 gdb 结合 qemu调试内核模块如果你正在使用模拟环境（如 qemu），可以使用 gdb 直接调试运行在 qemu 上的内核和模块。这种方式适合在没有真实硬件的情况下进行内核开发和调试。

kgdb 是 Linux 内核中的调试器，类似于用户态程序的 gdb，可以实现内核空间中的单步调试、断点设置和查看变量等功能。需要将内核编译为支持 kgdb 的版本，并进行远程调试。

步骤：

内核配置：配置内核使其支持 kgdb，需要开启以下选项：

CONFIG_KGDB=y
CONFIG_DEBUG_INFO=y
CONFIG_FRAME_POINTER=y

连接目标系统：通过串口或网络连接两个系统，一个用于运行被调试的内核，另一个用于运行 gdb。

启动调试会话：使用 gdb 连接目标系统后，可以像调试用户态程序一样进行调试。

优点：

支持单步执行、断点设置、查看变量等。
强大且全面的调试工具，适合复杂的内核调试。

缺点：

设置复杂，需要为内核配置和启动远程调试环境。
需要额外的硬件资源（如串口连接或远程调试器）。

可以参考这篇，虽然目前还没写完：VSCode调试Linux内核（TODO）-CSDN博客

5 使用 ftrace, strace 调试

ftrace 是 Linux 内核自带的跟踪工具，适合分析函数调用的轨迹。可以通过 ftrace 捕获内核模块中的函数执行流程，类似于函数级别的单步跟踪。

使用方法：
启用 ftrace 功能：
在 /sys/kernel/debug/tracing/ 目录中可以查看 ftrace 输出。
设置你要追踪的函数（比如你的模块中的函数）：
bash

复制代码

echo function_name > /sys/kernel/debug/tracing/set_ftrace_filter
开启追踪：
bash

复制代码

echo function > /sys/kernel/debug/tracing/current_tracer
查看追踪结果：
bash

复制代码

cat /sys/kernel/debug/tracing/trace
优点：

非侵入式调试，实时跟踪内核中函数的执行情况。
支持细粒度的跟踪，如特定函数调用、上下文切换等。

缺点：

不能直接像 gdb 那样设置断点和单步调试。
主要用于函数调用跟踪，调试信息较为简略。

这里有一个strace的例子，用的是LDD3的第一个示例程序。

pi@raspberrypi:~ $ strace ls /dev > /dev/scull0
execve("/usr/bin/ls", ["ls", "/dev"], 0x7eb695b4 /* 23 vars */) = 0
brk(NULL)                               = 0x10b4000
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76f3b000
access("/etc/ld.so.preload", R_OK)      = 0
openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=54, ...}) = 0
mmap2(NULL, 54, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0x76f3a000
close(3)                                = 0
readlink("/proc/self/exe", "/usr/bin/ls", 4096) = 11
openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\254\3\0\0004\0\0\0"..., 512) = 512
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=17708, ...}) = 0
mmap2(NULL, 147500, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76f15000
mmap2(0x76f20000, 81964, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x76f20000
munmap(0x76f15000, 45056)               = 0
munmap(0x76f35000, 16428)               = 0
mprotect(0x76f24000, 61440, PROT_NONE)  = 0
mmap2(0x76f33000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x76f33000
close(3)                                = 0
munmap(0x76f3a000, 54)                  = 0
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=30087, ...}) = 0
mmap2(NULL, 30087, PROT_READ, MAP_PRIVATE, 3, 0) = 0x76f18000
close(3)                                = 0
openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libselinux.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\0\0\0\0004\0\0\0"..., 512) = 512
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=198144, ...}) = 0
mmap2(NULL, 334964, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76ec6000
mmap2(0x76ed0000, 269428, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x76ed0000
munmap(0x76ec6000, 40960)               = 0
munmap(0x76f12000, 23668)               = 0
mprotect(0x76ef5000, 106496, PROT_NONE) = 0
mmap2(0x76f0f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2f000) = 0x76f0f000
mmap2(0x76f11000, 3188, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x76f11000
close(3)                                = 0
openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\20\346\1\0004\0\0\0"..., 512) = 512
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=1508068, ...}) = 0
mmap2(NULL, 1541036, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d57000
mmap2(0x76ec3000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16c000) = 0x76ec3000
mmap2(0x76ec6000, 37804, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x76ec6000
close(3)                                = 0
openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libpcre2-8.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\0\0\0\0004\0\0\0"..., 512) = 512
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=591716, ...}) = 0
mmap2(NULL, 721500, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76ca6000
mmap2(0x76cb0000, 655964, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x76cb0000
munmap(0x76ca6000, 40960)               = 0
munmap(0x76d51000, 21084)               = 0
mprotect(0x76d3f000, 65536, PROT_NONE)  = 0
mmap2(0x76d4f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8f000) = 0x76d4f000
close(3)                                = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76f39000
set_tls(0x76f39540)                     = 0
set_tid_address(0x76f390a8)             = 3169
set_robust_list(0x76f390ac, 12)         = 0
rseq(0x76f39520, 0x20, 0, 0xe7f5def3)   = 0
mprotect(0x76ec3000, 8192, PROT_READ)   = 0
mprotect(0x76d4f000, 4096, PROT_READ)   = 0
mprotect(0x76f0f000, 4096, PROT_READ)   = 0
mprotect(0x76f33000, 4096, PROT_READ)   = 0
mprotect(0x3f000, 4096, PROT_READ)      = 0
mprotect(0x76f64000, 4096, PROT_READ)   = 0
ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
munmap(0x76f18000, 30087)               = 0
statfs64("/sys/fs/selinux", 88, 0x7eca5428) = -1 ENOENT (No such file or directory)
statfs64("/selinux", 88, 0x7eca5428)    = -1 ENOENT (No such file or directory)
getrandom("\xba\xfb\xa8\x5e", 4, GRND_NONBLOCK) = 4
brk(NULL)                               = 0x10b4000
brk(0x10d5000)                          = 0x10d5000
openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0444, stx_size=0, ...}) = 0
read(3, "nodev\tsysfs\nnodev\ttmpfs\nnodev\tbd"..., 1024) = 388
read(3, "", 1024)                       = 0
close(3)                                = 0
access("/etc/selinux/config", F_OK)     = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=3048976, ...}) = 0
mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0x76ab0000
mmap2(NULL, 2596864, PROT_READ, MAP_PRIVATE, 3, 0x6f000) = 0x76836000
close(3)                                = 0
ioctl(1, TCGETS, 0x7eca5340)            = -1 ENOTTY (Inappropriate ioctl for device)
statx(AT_FDCWD, "/dev", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_MODE, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=STATX_ATTR_MOUNT_ROOT, stx_mode=S_IFDIR|0755, stx_size=3900, ...}) = 0
openat(AT_FDCWD, "/dev", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_CLOEXEC|O_DIRECTORY) = 3
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=STATX_ATTR_MOUNT_ROOT, stx_mode=S_IFDIR|0755, stx_size=3900, ...}) = 0
getdents64(3, 0x10b8bf8 /* 195 entries */, 32768) = 5784
getdents64(3, 0x10b8bf8 /* 0 entries */, 32768) = 0
close(3)                                = 0
statx(1, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFCHR|0777, stx_size=0, ...}) = 0
ioctl(1, TCGETS, 0x7eca2ee8)            = -1 ENOTTY (Inappropriate ioctl for device)
write(1, "autofs\nblock\nbtrfs-control\nbus\nc"..., 1206) = 1206
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

也可以参考这篇，也是还没写完：Linux的Strace（TODO）-CSDN博客

6 使用 kprobes 和 eBPF

kprobes 允许你动态地插入探测点到内核函数中，从而跟踪或调试某些函数的执行。结合 eBPF，你可以创建非常灵活的内核调试和性能分析工具。

优点：

动态插入探测点，不需要重新编译或重启内核。
结合 eBPF 后可以做到低开销的实时调试。

缺点：

需要一定的学习成本。
调试过程不如 kgdb 直接。

可以试一下这篇，继续没写完：eBPF试一下（TODO）-CSDN博客

参考：

万字长文，汇总 Linux 内核调试的方法