Linux 5.10 Pstore 功能测试
简介
PStore(Persistent store support)是用于系统发生oops或panic时,自动保存内核log buffer中的日志。随着功能不断完善,Duo S使用Linux 5.10已经支持保存console日志、ftrace消息和用户空间日志的收集,也支持将这些消息保存在不同的存储设备中,如内存、块设备或mtd设备。 为了提高灵活性和可扩展性,pstore将以上功能分别抽象为前端和后端,区别如下:
-
pstore
的前端,是指转存的日志类型,目前支持以下几个前端:- dmesg:主要是转存
Panic/Oops
时log_buf
里面的内核日志 - pmsg:提供给用户空间存储日志的入口,在Android里有看到被用于存储系统的日志。
- console:终端日志
- ftrace:
function trace
的信息
- dmesg:主要是转存
-
pstore
的后端,是指转存到什么类型的设备,目前支持以下几种后端:-
pstore/ram:Persistent Ram,重启不会丢数据的内存
-
pstore/blk:(v5.8以后的版本)所有可写的块设备,例如磁盘、U盘、emmc、NFTL nand等
-
mtd device:(v5.8以后的版本)mtd设备,例如 mtd nand。(mtd设备的支持依赖于 pstore/blk 后端,准确来说不是一种独立后端)
-
环境配置
source build/milkvsetup.sh
# Duo S
source device/milkv-duos-sd/boardconfig.sh
defconfig cv1813h_milkv_duos_sd
内核配置
修改配置文件:build/boards/cv181x/cv1813h_milkv_duos_sd/linux/cvitek_cv1813h_milkv_duos_sd_defconfig
diff --git a/build/boards/cv181x/cv1813h_milkv_duos_sd/linux/cvitek_cv1813h_milkv_duos_sd_defconfig b/build/boards/cv181x/cv1813h_milkv_duos_sd/linux/cvitek_cv1813h_milkv_duos_sd_defconfig
index d07358d26..c1475d639 100644
--- a/build/boards/cv181x/cv1813h_milkv_duos_sd/linux/cvitek_cv1813h_milkv_duos_sd_defconfig
+++ b/build/boards/cv181x/cv1813h_milkv_duos_sd/linux/cvitek_cv1813h_milkv_duos_sd_defconfig
@@ -530,3 +530,7 @@ CONFIG_BT_HCIUART_H4=y
# CONFIG_BT_HCIVHCI is not set
# CONFIG_BT_MRVL is not set
# CONFIG_BT_MTKSDIO is not set
+
+CONFIG_MAGIC_SYSRQ=y # sysRq 测试用
+
+CONFIG_PSTORE=y # PStore 功能开启
+CONFIG_PSTORE_PMSG=y # 前端:pmsg
+CONFIG_PSTORE_CONSOLE=y # 前端:console
+CONFIG_PSTORE_RAM=m # 后端:ram
说明:
CONFIG_PSTORE=y
,PStore 功能开启;CONFIG_PSTORE_CONSOLE=y
,PStore 前端:console;CONFIG_PSTORE_RAM=m
,PStore 后端:ram;
修改后使用如下命令使修改生效并重新生成新uImage
# 编译
build_kernel
报错:flex: not found
make[3]: *** [scripts/Makefile.lib:223:scripts/kconfig/zconf.tab.c] 错误 127
make[3]: *** 正在等待未完成的任务....
LEX scripts/kconfig/zconf.lex.c
/bin/sh: 1: flex: not found
确认为环境问题(flex工具缺失),切换为之前Docker搭建的编译环境中重新执行,编译正常。
Pstore模块
Pstore/ram需要编译成模块,这样加载时使用附带参数,方便调试。
模块参数
查找pstore模块源码,可以查找到如下可附带的参数:
gaoyang3513@SER7-Ubuntu22:~/Source/06-SG20x/02-Projects/SDK_SG200x_DuoS$ grep -rn "module_param" linux_5.10/fs/pstore/
linux_5.10/fs/pstore/ram.c:30:module_param(record_size, ulong, 0400);
linux_5.10/fs/pstore/ram.c:35:module_param_named(console_size, ramoops_console_size, ulong, 0400);
linux_5.10/fs/pstore/ram.c:39:module_param_named(ftrace_size, ramoops_ftrace_size, ulong, 0400);
linux_5.10/fs/pstore/ram.c:43:module_param_named(pmsg_size, ramoops_pmsg_size, ulong, 0400);
linux_5.10/fs/pstore/ram.c:47:module_param_hw(mem_address, ullong, other, 0400);
linux_5.10/fs/pstore/ram.c:52:module_param(mem_size, ulong, 0400);
linux_5.10/fs/pstore/ram.c:57:module_param(mem_type, uint, 0400);
linux_5.10/fs/pstore/ram.c:62:module_param_named(max_reason, ramoops_max_reason, int, 0400);
linux_5.10/fs/pstore/ram.c:67:module_param_named(ecc, ramoops_ecc, int, 0400);
linux_5.10/fs/pstore/ram.c:74:module_param_named(dump_oops, ramoops_dump_oops, int, 0400);
linux_5.10/fs/pstore/platform.c:44:module_param_named(update_ms, pstore_update_ms, int, 0600);
linux_5.10/fs/pstore/platform.c:80:module_param(backend, charp, 0444);
linux_5.10/fs/pstore/platform.c:89:module_param(compress, charp, 0444);
linux_5.10/fs/pstore/blk.c:22:module_param(kmsg_size, long, 0400);
linux_5.10/fs/pstore/blk.c:26:module_param(max_reason, int, 0400);
linux_5.10/fs/pstore/blk.c:35:module_param(pmsg_size, long, 0400);
linux_5.10/fs/pstore/blk.c:43:module_param(console_size, long, 0400);
linux_5.10/fs/pstore/blk.c:51:module_param(ftrace_size, long, 0400);
linux_5.10/fs/pstore/blk.c:55:module_param(best_effort, bool, 0400);
linux_5.10/fs/pstore/blk.c:83:module_param_string(blkdev, blkdev, 80, 0400);
其中,参考内核文档:,几个关键参数解读如下:
-
mem_address
,申请用于保存日志内存的起始地址; -
mem_size
,申请用于保存日志内存的大小;单位:Byte;要求:2幂值;
-
mem_type
,申请用于保存日志内存的类型;- 0(默认),pgprot_writecombine;
- 1,pgprot_noncached;
-
max_reason
,指定何种日志纳入管理(参考内核头文件:include/linux/kmsg_dump.h,枚举类型:enum kmsg_dump_reason);- 0 (KMSG_DUMP_UNDEF),由内核引导参数bootargs中的
printk.always_kmsg_dump
变量决定; - 1 (KMSG_DUMP_PANIC),仅内核Panic错误;
- 2 (KMSG_DUMP_OOPS),Oopse 和 Panic错误;
- 0 (KMSG_DUMP_UNDEF),由内核引导参数bootargs中的
日志存储
-
路径
实际在执行
mount
命令时可指定日志保存路径,如下:mount -t pstore pstore /sys/fs/pstore
说明:
-t pstore
,文件系统类型:pstore;/sys/fs/pstore
,挂载路径,即日志存储的位置;
-
文件名
参考文档:
linux_5.10/Documentation/admin-guide/ramoops.rst
中章节Reading the data
,可知文件命名格式如下:# 命名格式 <Front_End>_<Back_End>_<Record_Number>
说明:
-
<Front_End>
Pstore前端,即转存的日志类型; -
<Back_End>
Pstore后端,即转存到介质的类型; -
<Record_Number>
Pstore记录号。系统启动后,从0开始编号,每次触发一次关键事件,加1,最大值换算如下:MEM_SIZE >= RECORD_SIZE * n + CONSOLE_SIZE + FTRACE_SIZE + PMSG_SIZE
-
测试准备
固件升级
关于Duo S 如何网络升级固件,参考之前的文章《网络升级固件》。
版本验证
启动日志确认版本信息,以编译时间信息为依据:
[ 0.000000] Linux version 5.10.4-tag- (gaoyang3513@4c4f2694e4f0) (riscv64-unknown-linux-musl-gcc (Xuantie-900 linux-5.10.4 musl gcc Toolchain V2.6.1 B-20220906) 10.2.0, GNU ld (GNU Binutils) 2.35) #1 PREEMPT Tue Apr 9 11:28:02 CST 2024
资源更新
服务器更新TFTP根目录资源
cp -rf osdrv/extdrv/testing/oops.ko ~/tftp/
cp -rf ./linux_5.10/build/cv1813h_milkv_duos_sd/fs/pstore/ramoops.ko ~/tftp/
cp -rf linux_5.10/build/cv1813h_milkv_duos_sd/lib/reed_solomon/reed_solomon.ko ~/tftp/
cp -rf ./install/soc_cv1813h_milkv_duos_sd/rawimages/boot.sd ~/tftp/
Duo S TFTP下载并更新资源
# 镜像更新
# 创建boot目录并挂载至目录:/mnt/boot
mkdir -p /mnt/boot/ && mount /dev/mmcblk0p1 /mnt/boot/
# TFTP下载
#tftp -b 65536 -gr oops.ko -l /mnt/system/ko/oops.ko 192.168.8.108
tftp -b 65536 -gr ramoops.ko -l /mnt/system/ko/ramoops.ko 192.168.3.120
tftp -b 65536 -gr reed_solomon.ko -l /mnt/system/ko/reed_solomon.ko 192.168.3.120
tftp -b 65535 -gr boot.sd -l /mnt/boot/boot.sd 192.168.3.120
内存选取
由章节《模块参数》可知,ramoops模块加载需要附带参数,需要先确定:申请用于保存日志的内存的起始位置与大小?
-
使用节点
/proc/iomem
查看DRAM信息[root@milkv-duo]~# cat /proc/iomem | grep "System RAM" 80000000-9fdfffff : System RAM
-
使用
devmem
工具确认预申请的内存是否合法[root@milkv-duo]~# devmem 0x9fd00000 8 0x100000 [root@milkv-duo]~#
说明:
0x9fd00000
,选用系统内存最后1MB空间,即:0x09FD_0000~0x09FD_FFFF;8 0x100000
,位宽:8Bit,大小:0x10_0000(1MB);- devmem工具无报错,即验证可用;
使用如上内存信息佟大为加载ramoops模块时的参数,等到如下命令:
insmod /mnt/system/ko/ramoops.ko mem_address=0x9FD00000 mem_size=0x100000
报错:
[root@milkv-duo]~# insmod /mnt/system/ko/ramoops.ko mem_address=0x09FD0000 mem_size=0x100000
insmod: can't insert '/mnt/system/ko/ramoops.ko': unknown symbol in module, or unknown parameter
[root@milkv-duo]~# dmesg |tail
[ 972.359876] ramoops: Unknown symbol init_rs_gfp (err -2)
[ 972.359954] ramoops: Unknown symbol free_rs (err -2)
[ 976.406145] ramoops: Unknown symbol encode_rs8 (err -2)
[ 976.406188] ramoops: Unknown symbol decode_rs8 (err -2)
[ 976.406272] ramoops: Unknown symbol init_rs_gfp (err -2)
[ 976.406349] ramoops: Unknown symbol free_rs (err -2)
[ 976.415715] ramoops: Unknown symbol encode_rs8 (err -2)
[ 976.415759] ramoops: Unknown symbol decode_rs8 (err -2)
[ 976.415843] ramoops: Unknown symbol init_rs_gfp (err -2)
[ 976.415918] ramoops: Unknown symbol free_rs (err -2)
查看模块依赖关系文件:linux_5.10/build/cv1813h_milkv_duos_sd/modules/lib/modules/5.10.4-tag-/modules.dep
kernel/fs/pstore/ramoops.ko: kernel/lib/reed_solomon/reed_solomon.ko
说明:ramoops.ko依赖于reed_solomon.ko,所以加载顺序上先reed_solomon再ramoops;
整理后的的完整命令:
# 调试系统日志等级
echo 7 > /proc/sysrq-trigger
# 挂载pstore文件系统
mount -t pstore pstore /sys/fs/pstore
# 加载Pstore与依赖模块
insmod /mnt/system/ko/reed_solomon.ko
insmod /mnt/system/ko/ramoops.ko mem_address=0x9FD00000 mem_size=0x100000 dump_oops=1 mem_type=1 max_reason=2
触发Panic
尝试使用sysRq提供的MAGIC人为触发内核Panic,命令如下:
# Panic发生3S后重启
echo 3 > /proc/sys/kernel/panic
# 触发panic
echo c > /proc/sysrq-trigger
# 打印如下:
[ 2123.838214] sysrq: Trigger a crash
[ 2123.842502] Kernel panic - not syncing: sysrq triggered crash
[ 2123.855900] Rebooting in 3 seconds..
复位启动到系统后,再次挂载Pstore到/sys/fs/pstore
,很遗憾未没有见到异常捕获的日志:
[root@milkv-duo]~# ll /sys/fs/pstore/
total 0
dr-xr-xr-x 2 root root 0 Jan 1 00:02 ./
drwxr-xr-x 5 root root 0 Jan 1 00:02 ../
再次查看内核文档:linux_5.10/Documentation/admin-guide/ramoops.rst
,章节Setting the parameters
,可知有两种调试方式:
- bootargs中添加一个ramoops使用的
reserved-memory
内存对象; - DTS中添加一个名称固定为
ramoops
的reserved-memory
内存对象;
调试方案
方案一、Bootargs调试
Duo S有510MB系统内存可使用,参考章节《内存区域选取》在U-boot命令行下执行如下内容:
# 修改环境变量othbootargs,其被bootargs引用
setenv othbootargs 'earlycon=sbi riscv.fwsz=0x80000 loglevel=9 ramoops.mem_address=0x9FD00000 ramoops.mem_size=0x100000 ramoops.record_size=0x2000 ramoops.console_size=0x2000 memmap=0x100000$0x9FD00000'
# 参考U-boot下指导文档
setenv othbootargs 'earlycon=sbi riscv.fwsz=0x80000 loglevel=9 ramoops.mem_address=0x9FD00000 ramoops.mem_size=0x100000 memmap=0x100000$0x9FD00000'
# 引导启动
run bootcmd
环境初始化
# 调试系统日志等级
echo 7 > /proc/sysrq-trigger
# 挂载pstore文件系统
mount -t pstore pstore /sys/fs/pstore
# 加载Pstore与依赖模块
insmod /mnt/system/ko/reed_solomon.ko
insmod /mnt/system/ko/ramoops.ko mem_address=0x9FD00000 mem_size=0x100000 dump_oops=1 mem_type=1 max_reason=2
测试Panic
# Panic发生3S后重启
echo 3 > /proc/sys/kernel/panic
# 触发panic
echo c > /proc/sysrq-trigger
实现测试也没有成功,待排查。---- 24.04.09
猜测:可能Linux初始化引导参数解析阶段移除了ramoops的解析。
linux_5.10/init/main.c:142:char __initdata boot_command_line[COMMAND_LINE_SIZE];
方案二、DTS调试
diff --git a/build/boards/default/dts/cv181x/cv181x_default_memmap.dtsi b/build/boards/default/dts/cv181x/cv181x_default_memmap.dtsi
index bb362511c..d13fd0d61 100644
--- a/build/boards/default/dts/cv181x/cv181x_default_memmap.dtsi
+++ b/build/boards/default/dts/cv181x/cv181x_default_memmap.dtsi
@@ -21,5 +21,12 @@
compatible = "ion-region";
size = <0x0 CVIMMAP_ION_SIZE>;
};
+
+ ramoops@9fd00000 {
+ compatible = "ramoops";
+ reg = <0 0x9fd00000 0 0x100000>;
+ record-size = <0x4000>;
+ console-size = <0x4000>;
+ };
};
};
更新boot.sd
文件并重启
-
环境初始化
# 调试系统日志等级 echo 7 > /proc/sysrq-trigger # 挂载pstore文件系统 mount -t pstore pstore /sys/fs/pstore # 加载Pstore与依赖模块 insmod /mnt/system/ko/reed_solomon.ko insmod /mnt/system/ko/ramoops.ko
其中,ramoops加载时可不附带参数;
-
测试Panic
# Panic发生3S后重启 echo 3 > /proc/sys/kernel/panic # 触发panic echo c > /proc/sysrq-trigger
待设备重启
-
结果确认
重启完成后,再次环境初始化后,就可以在
/sys/fs/pstore
下见到捕获到的异常日志。[root@milkv-duo]~# ll /sys/fs/pstore total 0 drwxr-x--- 2 root root 0 Jan 1 00:02 ./ drwxr-xr-x 5 root root 0 Jan 1 00:02 ../ -r--r--r-- 1 root root 383 Jan 1 00:02 console-ramoops-0 -r--r--r-- 1 root root 16356 Jan 1 00:01 dmesg-ramoops-0
查看内容有:
[root@milkv-duo]~# cat /sys/fs/pstore/console-ramoops-0 |tail -n 20 [ 60.073528] pstore: Using crash dump compression: deflate [ 60.079312] printk: console [ramoops-1] enabled [ 60.084485] pstore: Registered ramoops as persistent store backend [ 60.091156] ramoops: using 0x100000@0x9fd00000, ecc: 0 [ 79.305649] sysrq: Trigger a crash [ 79.309903] Kernel panic - not syncing: sysrq triggered crash [ 79.319910] Rebooting in 3 seconds..
正常发生Panic时,系统最后的日志信息。
总结
- DTS调试成功,验证Pstore在Duo S(Linux 5.10)上,可以正常使用;
- Bootargs实现测试也没有成功,待排查。---- 24.04.09
扩展
参考U-boot文档:u-boot-2021.10/doc/usage/pstore.rst,在U-boot下支持Pstore分区内容的查看。
# pstore set <addr> <len> [record-size] [console-size] [ftrace-size] [pmsg_size] [ecc-size]
=> pstore set 0x9fd00000 0x100000 0x2000 0x2000
参考
-
参考1. linux_5.10/Documentation/admin-guide/ramoops.rst
-
参考2. linux_5.10/tools/testing/selftests/pstore/pstore_crash_test;
-
参考3. u-boot-2021.10/doc/usage/pstore.rst;
-
参考6. Linux内核调试之一个简单的oops
-
参考9. 【调试】pstore原理和使用方法总结
备注
-
reed_solomon更新
cp -rf linux_5.10/build/cv1813h_milkv_duos_sd/lib/reed_solomon/reed_solomon.ko ~/tftp/ # Duo S 下载 tftp -b 65536 -gr reed_solomon.ko -l /mnt/system/ko/reed_solomon.ko 192.168.8.108
-
启动信息
[ 0.114376] ramoops: using module parameters [ 0.121832] printk: console [ramoops-1] enabled [ 0.126680] pstore: Registered ramoops as persistent store backend [ 0.133024] ramoops: using 0x100000@0x9fd00000, ecc: 0