简介
系统好多次无缘无故的宕机了,还好第一次宕机的时候部署看最重要的kdump服务
接下来一起看看问题到底是什么引起的吧
日志
从生成的vmcore中获取到panic的日志,居然是Hardware Error错误
PANIC: "{1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0"
{1}[Hardware Error]:
这是一个硬件错误报告的标识符,表明系统检测到了一个硬件级别的问题。
{2}Hardware error from APEI:
APEI(ACPI Platform Error Interface)是一个由 ACPI(高级配置和电源接口)规范定义的接口,用于报告平台级别的硬件错误。它提供了一种标准化的方式来报告硬件错误,便于系统处理和日志记录。
{3}Generic Hardware Error Source: 0:
这表示错误来源的具体信息。在这里,“Generic Hardware Error Source: 0”说明错误来源被标记为“0”,通常这意味着错误来源是通用的或者没有特定的硬件标识符。错误代码“0”可能表示未指定或默认的错误来源。
这个错误报告表明系统检测到一个硬件错误,并且这个错误是通过 APEI 接口报告的。错误来源被标记为“0”
咱们进一步看看dmesglog的信息:
[862858.612710] {23}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 9
[862858.612711] {23}[Hardware Error]: event severity: recoverable
[862858.612712] {23}[Hardware Error]: Error 0, type: recoverable
[862858.612712] {23}[Hardware Error]: section_type: ARM processor error
[862858.612713] {23}[Hardware Error]: MIDR: 0x00000000481fd010
[862858.612713] {23}[Hardware Error]: Multiprocessor Affinity Register (MPIDR): 0x00000000810c0000
[862858.612714] {23}[Hardware Error]: error affinity level: 0
[862858.612714] {23}[Hardware Error]: running state: 0x1
[862858.612714] {23}[Hardware Error]: Power State Coordination Interface state: 0
[862858.612715] {23}[Hardware Error]: Error info structure 0:
[862858.612715] {23}[Hardware Error]: num errors: 1
[862858.612715] {23}[Hardware Error]: error_type: 0, cache error
[862858.612716] {23}[Hardware Error]: error_info: 0x0000000000400014
[862858.612716] {23}[Hardware Error]: cache level: 1
[862858.612717] {23}[Hardware Error]: the error has not been corrected
[862858.612717] {23}[Hardware Error]: virtual fault address: 0x0000000000000000
[862858.612718] {23}[Hardware Error]: physical fault address: 0x00000023719a1a40
[862858.612718] {23}[Hardware Error]: Vendor specific error info has 16 bytes:
[862858.612719] {23}[Hardware Error]: 00000000: 00000000 00000000 00000000 00000000 ................
1、此内报告了硬件错误,但是容貌似不会引起崩溃,严重等级是:recoverable(可恢复的,意味着系统可以在不需要重启的情况下处理这个错误)
2、提供的物理地址和虚拟地址可帮助进一步判断,小编因对硬件类不懂,咱们接着往下分析。
[862858.737186] UCE: kernel recovery 0, java-2133353 is user-thread.
[862858.743251] Internal error: Uncorrected hardware memory error in kernel-access
: 96000610 [#1] SMP
[862858.753722] Modules linked in: ext4 crc16 mbcache jbd2 fscrypto fuse st sr_mod cdrom cfg80211 rfkill sunrpc vfat fat dm_service_time dm_multipath ipmi_ssif ses enclosure aes_ce_blk crypto_simd cryptd aes_ce_cipher crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce sbsa_gwdt hns_roce_hw_v2 hns_roce ib_core sg ipmi_si ipmi_devintf ipmi_msghandler hisi_uncore_hha_pmu hisi_uncore_ddrc_pmu hisi_uncore_l3c_pmu spi_dw_mmio hisi_uncore_pmu binfmt_misc sch_fq_codel ip_tables realtek hclge sd_mod qla2xxx hisi_sas_v3_hw hisi_sas_main libsas ahci mlx5_core libahci scsi_transport_sas nvme_fc hns3 nfit nvme_fabrics hibmc_drm libata megaraid_sas nvme_core ttm mlxfw hnae3 libnvdimm devlink host_edma_drv scsi_transport_fc i2c_designware_platform i2c_designware_core dm_mirror dm_region_hash dm_log dm_mod
[862858.823158] Process java (pid: 2133353, stack limit = 0x00000000b447f901)
[862858.830003] CPU: 16 PID: 2133353 Comm: java Kdump: loaded Not tainted 4.19.90-2305.1.0.0199.78.uel20.aarch64 #1
[862858.840128] Hardware name: Huawei S920X00K/BC82AMDDIA, BIOS 1.86.K 01/10/2022
[862858.847315] pstate: 20c00009 (nzCv daif +PAN +UAO)
[862858.852175] pc : __memcpy_mc+0x114/0x150
[862858.856168] lr : iter_memcpy_mc+0x24/0x30
[862858.860244] sp : ffffa03220627740
[862858.863628] x29: ffffa03220627740 x28: ffffa0301420e450
[862858.869002] x27: 0000000000000000 x26: ffff802092020000
[862858.874376] x25: ffffa032206278b0 x24: ffffa0301420e400
[862858.879750] x23: 0000000000000000 x22: ffffa03220627878
[862858.885122] x21: 0000000000010000 x20: 0000000000010000
[862858.890495] x19: 0000000000010000 x18: 0000000000000000
[862858.895867] x17: 0000000000000000 x16: 0000000000000000
[862858.901239] x15: 0000fffd437ee980 x14: 00000000bb3e8366
[862858.906613] x13: bb3e835fbb3e8348 x12: a613a8920001b868
[862858.911987] x11: 000000000000000d x10: 0000000000000000
[862858.917360] x9 : bb3e40fb0009f610 x8 : 000000000000000d
[862858.922733] x7 : 00000000bb3e8341 x6 : ffff802092011a10
[862858.928106] x5 : 0000000000000000 x4 : 0000000000000000
[862858.933480] x3 : 0000000001400740 x2 : 000000000000e580
[862858.938853] x1 : ffff8023719a1a40 x0 : ffff802092010000
1、此报告提示系统检测到一个未纠正的硬件内存错误。这种错误通常表示物理内存条或其他硬件组件出现了严重的问题。
2、崩溃时正在运行的进程java-2133353
3、pc : __memcpy_mc+0x114/0x150
崩溃时程序计数器(PC寄存器)指向 __memcpy_mc 函数内,这个函数通常涉及内存复制操作。
4、详细的服务器相关信息并也给出了BIOS的信息:Hardware name: Huawei S920X00K/BC82AMDDIA, BIOS 1.86.K 01/10/2022
5、上述问题可以判断,这个内核崩溃是由硬件内存错误引起的。这种错误通常需要进行硬件检查,例如检查内存条是否安装正确,或者更换内存条。
内核崩溃的详细栈堆:
[862858.944228] Call trace:
[862858.946750] __memcpy_mc+0x114/0x150
[862858.950396] _copy_from_iter+0x2a4/0x388
[862858.954387] copy_page_from_iter+0x15c/0x208
[862858.958724] pipe_write+0xcc/0x440
[862858.962199] __vfs_write+0x148/0x1c0
[862858.965845] __kernel_write+0x84/0x168
[862858.969664] dump_emit+0xb4/0x100
[862858.973050] elf_core_dump+0x86c/0xa50
[862858.976868] do_coredump+0x4e8/0xe70
[862858.980515] get_signal+0x5f4/0x868
[862858.984075] do_signal+0x8c/0x298
[862858.987460] do_notify_resume+0xf8/0x360
[862858.991450] work_pending+0x8/0x10
[862858.994924] Code: a8c12829 a8c1302b a8c1382d a88120c7 (a8c12027)
[862859.001119] SMP: stopping secondary CPUs
[862859.008902] Starting crashdump kernel...
[862859.012894] Bye!
崩溃发生在 __memcpy_mc 函数中,可能与内存复制相关。对应了上述发生的问题。
内存复制是什么?
指的是将内存中的数据从一个位置复制到另一个位置。这种操作可以涉及多个方面,例如:
1、内存管理:在操作系统或应用程序中,内存复制通常用于管理内存块的分配和释放,确保数据在不同内存区域之间正确传递。
2、数据处理:程序在处理数据时,如读写文件或网络通信时,可能需要将数据从一个缓冲区复制到另一个缓冲区。
3、系统调用:内核和用户空间之间的系统调用可能会涉及内存复制,例如在执行 I/O 操作时。