Zephyr 是一个开源、可扩展、适应性强的实时操作系统 (RTOS),能够在多种硬件平台上运行,包括 乐鑫 的 ESP32、ESP32-S2、ESP32-C3 和 ESP32-S3 系列产品。Zephyr 为嵌入式系统开发提供了丰富的功能,包括针对无法恢复的软件错误生成可供分析的 core dump。
Core dump 是在程序崩溃时保留的内存快照,可用于调试程序,查找崩溃原因、以及确定导致崩溃的具体任务、代码行和调用堆栈。如果需要,还可以为变量赋值并存储变量内容。
Zephyr 操作系统支持多个 core dump 后端,其中包括日志后端。日志后端将 core dump 输出到 UART,然后保存到文件中,这样就可以使用自定义的 GDB 服务器和 SDK 提供的 GDB 进行分析。
如何使用
要使用 core dump 的日志后端,需要启用相应的 Kconfig 选项:
CONFIG_DEBUG_COREDUMP=y
CONFIG_DEBUG_COREDUMP_BACKEND_LOGGING=y
一旦启用了日志后端,应用程序在遇到致命错误时将生成 core dump。CPU 寄存器和内存内容将被打印到控制台。
将内容复制并粘贴到一个名为 coredump.log 的文件中。
我们需要将这个文本文件转换为自定义 GDB 服务器可以解析的二进制文件。只需运行 coredump_serial_log_parser.py 脚本即可。
./scripts/coredump/coredump_serial_log_parser.py coredump.log coredump.bin
该脚本会把二进制文件输出到 coredump.bin。
使用 zephyr 应用程序中的 .elf 文件,并将 coredump.bin 作为参数,启动自定义的 GDB 服务器:
./scripts/coredump/coredump_gdbserver.py build/zephyr/zephyr.elfcoredump.bin
打开另一个终端,从 Zephyr SDK 里运行 Xtensa 的 GDB,将 .elf 文件作为参数:
~/zephyr-sdk-0.16.0/xtensa-espressif_esp32_zephyr-elf/bin/xtensa-espressif_esp32_zephyr-elf-gdb build/zephyr/zephyr.elf
在 GDB 内部,使用以下命令连接到自定义服务器:
(gdb) target remote localhost:1234
现在,您可以检查程序崩溃时的状态,读取变量值、回溯信息和寄存器值了。
示例
下面是一个使用 core dump 进行调试的例子。
我们在 tests/subsys/debug/coredump 里创建并烧录 core dump 示例。
该示例定义了三个函数:func_1,func_2 和 func_3。func_1 调用 func_2,func_2 调用func_3。func_3 试图对空指针进行解引用,这将导致程序崩溃。
程序崩溃会生成一个 core dump 文件,我们将用它来调试程序。
1.创建应用程序:
west build -p -b esp32 tests/subsys/debug/coredump
2.烧录并监控 ESP32 开发板:
west flash && west espressif monitor
预期输出:
*** Booting Zephyr OS build zephyr-v3.3.0-3986-gebf86941118f ***
Coredump: esp32
E: ** FATAL EXCEPTION
E: ** CPU 0 EXCCAUSE 29 (store prohibited)
E: ** PC 0x400d0435 VADDR (nil)
E: ** PS 0x60620
E: ** (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:6 CALLINC:2)
E: ** A0 0x80081716 SP 0x3ffe65e0 A2 0x3f401cb8 A3 (nil)
E: ** A4 0x3f401cb8 A5 0xff A6 0x3ffb1ad8 A7 0x3ffe5dfc
E: ** A8 (nil) A9 0x3ffe6590 A10 0x3f400968 A11 0x3f4012d0
E: ** A12 0x3ffb1ad8 A13 0x60420 A14 0x3ffe5dd8 A15 0x3ffe5e30
E: ** LBEG (nil) LEND (nil) LCOUNT (nil)
E: ** SAR 0x1b
E: #CD:BEGIN#
E: #CD:5a4501000500050000000000
E: #CD:4101006800
E: #CD:0202000135040d401d000000000000001b000000200606000000000016170880
E: #CD:e065fe3fb81c403f00000000b81c403fff000000d81afb3ffc5dfe3f00000000
E: #CD:9065fe3f6809403fd012403f000000009065fe3f6809403fd012403f00000000
E: #CD:0000000000000000
E: #CD:4d0100d81afb3f581bfb3f
E: #CD:d81cfb3fd81cfb3f000000000180000000000000000000000000000000000000
E: #CD:000000000000000000000000000000000000000000000000101bfb3f101bfb3f
E: #CD:6d61696e00000000000000000000000000000000000000000000000000000000
E: #CD:00000000305efe3f0008000000000000f5ffffff0864fe3f080afb3fffffffff
E: #CD:4d0100305efe3f3066fe3f
E: #CD:f4b6dbfcd7903c37a5f85ffb73a1c5ad7da3432d420e03142b7673dc1f70a451
E: #CD:e4165f6f2491c43b0aae041aa6bad45d0d019d7bd7061165478aa8e8e9dfb05d
E: #CD:5e9ca79574bd8eee1a446b5919d39bbdc7a78c067dd9348e7dd99f3bc0ae0e59
E: #CD:05a5d5e1459c6325d537dd4db75d0210f30e67e7e292f5af90ea8bab11d9090c
E: #CD:14bacecb11dc47c19ae8d3922a290fcbe82ff95e23d63ad210459e8fc79dc78a
E: #CD:e443c9dbf22b94b0ed95e23bba0f6006022225d48e85ba76d148caff2b2fd519
E: #CD:324d6de3d17410eeae3d5a6513bfc8eed68141495be8c069e69dcea9748fd38e
E: #CD:59fc1f6e57f36d52ba51d383d2df4ab9746d893b9bf276879d2182cfd03fe1fe
E: #CD:fce68691b4df96c5ac234d34a8eb21f199191138061a17245042ab8ac94e14ac
E: #CD:0e017619e4b42895aec67da7fdfb525168850547d0f2530247b1995dfaf9ddc5
E: #CD:168588d974a379bcc3ddbead2edba5571dcbdd29c06ab11b41793d3ba2e15ab4
E: #CD:da808c516438cd28215077c160bb5e0221b6c2d827db238850687d5cad9736f4
E: #CD:60f9f777da01758a56b11a2fa9fc4bd9eaddb172f844223ea356ac0d5f77324f
E: #CD:14417a13726f99d28c42bdedf33cc9a198dbe361b4974d152256806c0db076f8
E: #CD:12b3e2c95e5dc84dd6d028065abf26054ff992b205ca641e249417dd7368bd92
E: #CD:d8faf9e19687346bf31e315db64a05c049b23cf3a10eb60d6472211fdd43e8cc
E: #CD:7b334dbb3b549c5b91bea3352ac68b3f6b199e2967344cc07da7d79c4bdc884c
E: #CD:6ebcd3093df87d7633ccb583b036864cc48166283095024d592b16600f4ed38f
E: #CD:bd1e1bdaf1538c6134c078d1f7b69f9467774e910b318854a0fcdd8841b12e0d
E: #CD:74ca51cad7c6b020789c6affacc31d39a034d13c0ebaa3a07b53f81091d7b704
E: #CD:8641f161cbaacd75c8aa1bd1b557470083e87921b4ae92960764aa2ad6eff203
E: #CD:c32fa391df51c8c38aa88c5a54bf130009f26b64a8b2a15418b4286ab0edc3ec
E: #CD:08d9c9c133fad8213c38e9ccb7505e2940c9612ceeb772440786012cddf5bbaf
E: #CD:435c265247afa7e2cdb315fa23e47a45b9050d8cb61b4a72646f58febc81773f
E: #CD:cbf027958c8a67478a2f9441bc94767ec90f446ba54ff82b893dddaddd42a004
E: #CD:2a26cb016dfb2c1e0c691e319505c4167e83031ee83cd0843165c00b63fa4e96
E: #CD:01f75bce679271064f56595a83ca5ae00ea79ba5c396f560a709dfa617e1dd14
E: #CD:714ee1c87b95a422386d41a70f88df76019315875cc82b030dd005ba35af53bc
E: #CD:029df20768fe4612c91133a5f38602715c05ef9b330ea662402b4779b8de6e8b
E: #CD:6d8cad92e2fe7600ea3c4c83744186abc127d75f6f9334365e0c6b8c2ea6c82b
E: #CD:7ea6b0095adbdbc82ba150a670db718a67d5fe5d8b9f04a48eafff8d2200e075
E: #CD:d18277cc7bcbd8419bb21b15cfddc8340bd7721707bd54ff04fb72c71ec12ce7
E: #CD:8ce9635cf260717cbe2cf8392f7796c24a2384dd9db2142dc00e7dc83c9b8119
E: #CD:3e8a971d612a603e88062e7679ada4082e8fddc80cdd34bfd9df64f4a887d9b0
E: #CD:920fff4ae8ee63e1e48e611e887334929a89ccce5ecb250c03ca59178b3eb5dc
E: #CD:d88de936c368020f892357c7700d2b9ac237affbe9aad454736025fc37e7a4f0
E: #CD:b60103116de8e418f17b1ffea7c3840ce9b5de74f2714451e7c774b0fc94d2d1
E: #CD:8c3e9f17ac30161adf79c9cee52cfa79e62cc228781234124557f6463fe95228
E: #CD:80d4d0b905bc05fb54e134ced7f84ac128ba82cda103f29f9a62e4c5f4105021
E: #CD:80f085cb993e77c86f82ecf01dac257e25942f840eccd2e11385a946fd7b508c
E: #CD:71e13192d9b4ceba06ae199b77127b529791be9b878657590a3c5a263d9c8319
E: #CD:b9f9487e4086539f1e4758b87ca24b70887d185c7e617ff288bde4944e2c6cf3
E: #CD:fb23feb7aadf90c80c6d00f66d287f80f52c7d593a108e9ff178fc30defc40d4
E: #CD:3b59f4efbd708b0f0192d457450ad657d3b45fc4d254406d7ac94f4eafdc5be4
E: #CD:ba1719090cbed14af915ca7991db7b87027ced573a0e6aff7078a83139df611f
E: #CD:3fa23f95b5c39b7e15c92108a5322206fb74ee59e39327b66180c48dd9cbc1cb
E: #CD:00c04eeb3550aa7e24de68fa4aa720be1ee74e8dc10a57a93c64fe3f00000000
E: #CD:00c050eb0000000000000000703d088000c04eeb010000006d00000000000000
E: #CD:3266fe3f0000000000000000db32008c00c050eb010000000000000000000000
E: #CD:0a0000000000000020020600703d0840093d0880001008406d000000a01c403f
E: #CD:7c330880a064fe3f6d00000000000000fedf99ececff33aec26b3658ed9c5be5
E: #CD:283d0880c064fe3f6d00000006000000c01cfb3fd81afb3fd81afb3fd865fe3f
E: #CD:4c3d08804065fe3f6809403fc065fe3fd012403f000000008000037300000000
E: #CD:00000000000000001a2488bc891e7d358fc4ecfa81e597aa9df65bb7a6a74806
E: #CD:c065fe3fa065fe3f04000000c065fe3fa065fe3ffc3c0840000000007509403f
E: #CD:000000007209403fdb8a502716f5e7f50400000025000000a065fe3f04000000
E: #CD:30040d809065fe3f6809403fd012403f000000003b4bed4f08035fadadc14494
E: #CD:0a1751af3f39a63cf90f74b2820658d1cbb50403fb4800949c65fe3fd81afb3f
E: #CD:20040600d85dfe3f305efe3f000000009065fe3f6809403fd012403fb81c403f
E: #CD:ff000000d81afb3ffc5dfe3fdf2daf23000000001d0000000000000000000000
E: #CD:000000001b0000002006060035040d401617088000100840b81c403f00000000
E: #CD:56040d800066fe3f0000000000000000b81c403fff000000d81afb3ffc5dfe3f
E: #CD:000000002066fe3ff81608400000000000000000000000007f00000000000000
E: #CD:000000004066fe3f000000000000000000000000000000000000000000000000
E: #CD:END#
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x3ffb1ad8 (main)
E: Halting system
Coredump内容以#CD:BEGIN#开始,以#CD:END#结束。我们需要将这两者之间的内容复制到一个名为 coredump.log 的新文件中。
3.将 core dump 复制到 coredump.log:
E: #CD:BEGIN#
E: #CD:5a4501000500050000000000
E: #CD:4101006800
E: #CD:0202000135040d401d000000000000001b000000200606000000000016170880
E: #CD:e065fe3fb81c403f00000000b81c403fff000000d81afb3ffc5dfe3f00000000
E: #CD:9065fe3f6809403fd012403f000000009065fe3f6809403fd012403f00000000
E: #CD:0000000000000000
E: #CD:4d0100d81afb3f581bfb3f
E: #CD:d81cfb3fd81cfb3f000000000180000000000000000000000000000000000000
E: #CD:000000000000000000000000000000000000000000000000101bfb3f101bfb3f
E: #CD:6d61696e00000000000000000000000000000000000000000000000000000000
E: #CD:00000000305efe3f0008000000000000f5ffffff0864fe3f080afb3fffffffff
E: #CD:4d0100305efe3f3066fe3f
E: #CD:f4b6dbfcd7903c37a5f85ffb73a1c5ad7da3432d420e03142b7673dc1f70a451
E: #CD:e4165f6f2491c43b0aae041aa6bad45d0d019d7bd7061165478aa8e8e9dfb05d
E: #CD:5e9ca79574bd8eee1a446b5919d39bbdc7a78c067dd9348e7dd99f3bc0ae0e59
E: #CD:05a5d5e1459c6325d537dd4db75d0210f30e67e7e292f5af90ea8bab11d9090c
E: #CD:14bacecb11dc47c19ae8d3922a290fcbe82ff95e23d63ad210459e8fc79dc78a
E: #CD:e443c9dbf22b94b0ed95e23bba0f6006022225d48e85ba76d148caff2b2fd519
E: #CD:324d6de3d17410eeae3d5a6513bfc8eed68141495be8c069e69dcea9748fd38e
E: #CD:59fc1f6e57f36d52ba51d383d2df4ab9746d893b9bf276879d2182cfd03fe1fe
E: #CD:fce68691b4df96c5ac234d34a8eb21f199191138061a17245042ab8ac94e14ac
E: #CD:0e017619e4b42895aec67da7fdfb525168850547d0f2530247b1995dfaf9ddc5
E: #CD:168588d974a379bcc3ddbead2edba5571dcbdd29c06ab11b41793d3ba2e15ab4
E: #CD:da808c516438cd28215077c160bb5e0221b6c2d827db238850687d5cad9736f4
E: #CD:60f9f777da01758a56b11a2fa9fc4bd9eaddb172f844223ea356ac0d5f77324f
E: #CD:14417a13726f99d28c42bdedf33cc9a198dbe361b4974d152256806c0db076f8
E: #CD:12b3e2c95e5dc84dd6d028065abf26054ff992b205ca641e249417dd7368bd92
E: #CD:d8faf9e19687346bf31e315db64a05c049b23cf3a10eb60d6472211fdd43e8cc
E: #CD:7b334dbb3b549c5b91bea3352ac68b3f6b199e2967344cc07da7d79c4bdc884c
E: #CD:6ebcd3093df87d7633ccb583b036864cc48166283095024d592b16600f4ed38f
E: #CD:bd1e1bdaf1538c6134c078d1f7b69f9467774e910b318854a0fcdd8841b12e0d
E: #CD:74ca51cad7c6b020789c6affacc31d39a034d13c0ebaa3a07b53f81091d7b704
E: #CD:8641f161cbaacd75c8aa1bd1b557470083e87921b4ae92960764aa2ad6eff203
E: #CD:c32fa391df51c8c38aa88c5a54bf130009f26b64a8b2a15418b4286ab0edc3ec
E: #CD:08d9c9c133fad8213c38e9ccb7505e2940c9612ceeb772440786012cddf5bbaf
E: #CD:435c265247afa7e2cdb315fa23e47a45b9050d8cb61b4a72646f58febc81773f
E: #CD:cbf027958c8a67478a2f9441bc94767ec90f446ba54ff82b893dddaddd42a004
E: #CD:2a26cb016dfb2c1e0c691e319505c4167e83031ee83cd0843165c00b63fa4e96
E: #CD:01f75bce679271064f56595a83ca5ae00ea79ba5c396f560a709dfa617e1dd14
E: #CD:714ee1c87b95a422386d41a70f88df76019315875cc82b030dd005ba35af53bc
E: #CD:029df20768fe4612c91133a5f38602715c05ef9b330ea662402b4779b8de6e8b
E: #CD:6d8cad92e2fe7600ea3c4c83744186abc127d75f6f9334365e0c6b8c2ea6c82b
E: #CD:7ea6b0095adbdbc82ba150a670db718a67d5fe5d8b9f04a48eafff8d2200e075
E: #CD:d18277cc7bcbd8419bb21b15cfddc8340bd7721707bd54ff04fb72c71ec12ce7
E: #CD:8ce9635cf260717cbe2cf8392f7796c24a2384dd9db2142dc00e7dc83c9b8119
E: #CD:3e8a971d612a603e88062e7679ada4082e8fddc80cdd34bfd9df64f4a887d9b0
E: #CD:920fff4ae8ee63e1e48e611e887334929a89ccce5ecb250c03ca59178b3eb5dc
E: #CD:d88de936c368020f892357c7700d2b9ac237affbe9aad454736025fc37e7a4f0
E: #CD:b60103116de8e418f17b1ffea7c3840ce9b5de74f2714451e7c774b0fc94d2d1
E: #CD:8c3e9f17ac30161adf79c9cee52cfa79e62cc228781234124557f6463fe95228
E: #CD:80d4d0b905bc05fb54e134ced7f84ac128ba82cda103f29f9a62e4c5f4105021
E: #CD:80f085cb993e77c86f82ecf01dac257e25942f840eccd2e11385a946fd7b508c
E: #CD:71e13192d9b4ceba06ae199b77127b529791be9b878657590a3c5a263d9c8319
E: #CD:b9f9487e4086539f1e4758b87ca24b70887d185c7e617ff288bde4944e2c6cf3
E: #CD:fb23feb7aadf90c80c6d00f66d287f80f52c7d593a108e9ff178fc30defc40d4
E: #CD:3b59f4efbd708b0f0192d457450ad657d3b45fc4d254406d7ac94f4eafdc5be4
E: #CD:ba1719090cbed14af915ca7991db7b87027ced573a0e6aff7078a83139df611f
E: #CD:3fa23f95b5c39b7e15c92108a5322206fb74ee59e39327b66180c48dd9cbc1cb
E: #CD:00c04eeb3550aa7e24de68fa4aa720be1ee74e8dc10a57a93c64fe3f00000000
E: #CD:00c050eb0000000000000000703d088000c04eeb010000006d00000000000000
E: #CD:3266fe3f0000000000000000db32008c00c050eb010000000000000000000000
E: #CD:0a0000000000000020020600703d0840093d0880001008406d000000a01c403f
E: #CD:7c330880a064fe3f6d00000000000000fedf99ececff33aec26b3658ed9c5be5
E: #CD:283d0880c064fe3f6d00000006000000c01cfb3fd81afb3fd81afb3fd865fe3f
E: #CD:4c3d08804065fe3f6809403fc065fe3fd012403f000000008000037300000000
E: #CD:00000000000000001a2488bc891e7d358fc4ecfa81e597aa9df65bb7a6a74806
E: #CD:c065fe3fa065fe3f04000000c065fe3fa065fe3ffc3c0840000000007509403f
E: #CD:000000007209403fdb8a502716f5e7f50400000025000000a065fe3f04000000
E: #CD:30040d809065fe3f6809403fd012403f000000003b4bed4f08035fadadc14494
E: #CD:0a1751af3f39a63cf90f74b2820658d1cbb50403fb4800949c65fe3fd81afb3f
E: #CD:20040600d85dfe3f305efe3f000000009065fe3f6809403fd012403fb81c403f
E: #CD:ff000000d81afb3ffc5dfe3fdf2daf23000000001d0000000000000000000000
E: #CD:000000001b0000002006060035040d401617088000100840b81c403f00000000
E: #CD:56040d800066fe3f0000000000000000b81c403fff000000d81afb3ffc5dfe3f
E: #CD:000000002066fe3ff81608400000000000000000000000007f00000000000000
E: #CD:000000004066fe3f000000000000000000000000000000000000000000000000
E: #CD:END#
4.转换为二进制格式:
./scripts/coredump/coredump_serial_log_parser.py coredump.log coredump.bin
5.启动自定义的 GDB 服务器:
./scripts/coredump/coredump_serial_log_parser.py coredump.log coredump.bin
预期输出:
[INFO][gdbstub] Log file: coredump.bin
[INFO][gdbstub] ELF file: build/zephyr/zephyr.elf
[INFO][parser] Reason: K_ERR_CPU_EXCEPTION
[INFO][parser] Pointer size 32
[INFO][parser] Memory: 0x3ffb1ad8 to 0x3ffb1b58 of size 128
[INFO][parser] Memory: 0x3ffe5e30 to 0x3ffe6630 of size 2048
[INFO][parser] ELF Section: 0x0 to 0x1f of size 32 (read-only data)
[INFO][parser] ELF Section: 0x20 to 0x3b of size 28 (read-only data)
[INFO][parser] ELF Section: 0x40080000 to 0x400803ff of size 1024 (text)
[INFO][parser] ELF Section: 0x40080400 to 0x40083e7b of size 14972 (text)
[INFO][parser] ELF Section: 0x3f400040 to 0x3f401bff of size 7104 (read-only data)
[INFO][parser] ELF Section: 0x3f401c00 to 0x3f401c3f of size 64 (read-only data)
[INFO][parser] ELF Section: 0x3f401c40 to 0x3f401cb7 of size 120 (read-only data)
[INFO][parser] ELF Section: 0x3f401cb8 to 0x3f401cd9 of size 34 (read-only data)
[INFO][parser] ELF Section: 0x3ffb0a20 to 0x3ffb0a57 of size 56 (read-only data)
[INFO][parser] ELF Section: 0x400d0020 to 0x400d3d53 of size 15668 (text)
[INFO][gdbstub] Waiting GDB connection on port 1234...
6.打开一个新的终端,启动 Zephyr SDK 中的 Xtensa ESP32 GDB:
~/zephyr-sdk-0.16.0/xtensa-espressif_esp32_zephyr-elf/bin/xtensa-espressif_esp32_zephyr-elf-gdb build/zephyr/zephyr.elf
7.在 GDB 内部,连接到远程服务器:
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0x400d0435 in func_3 (addr=0x0) at zephyr/tests/subsys/debug/coredump/src/main.c:27
27 *addr = 0;
8.运行 bt 命令查看回溯信息:
(gdb) bt
#0 0x400d0435 in func_3 (addr=0x0)
at zephyr/tests/subsys/debug/coredump/src/main.c:27
#1 func_2 (addr=0x0) at zephyr/tests/subsys/debug/coredump/src/main.c:40
#2 func_1 (addr=0x0) at zephyr/tests/subsys/debug/coredump/src/main.c:45
#3 main () at zephyr/tests/subsys/debug/coredump/src/main.c:52
结论
Core dump 是一个强大的工具,用于调试 ESP32 开发板上的 Zephyr 程序错误。通过使用 core dump 的日志后端,您可以轻松生成 core dump 文件,并使用调试器进行分析。这将帮助您快速、方便地找出程序崩溃的原因,并修复 bug。