RHEL6: Server panicked in 'redirfs' module

环境

• Red Hat Enterprise Linux 6
• 3rd party [redirfs]

问题

• Actual panic message can vary significantly. 
  One of the examples of panic message is:

Raw

kernel BUG at /build/DEEPSECURITY_DSA/9.5_SP1/rhel6_64/en/Source/src/output/RedHat_EL6.x86_64/vfs-filter/2.6.32-431.el6.x86_64/3rd_party/redirfs-0.10/rfs_inode.c:114!

Another examples:

Raw

general protection fault: 0000 [#1] SMP 
last sysfs file: /sys/devices/virtual/block/dm-129/range
CPU 5 
Modules linked in: <long list of modules here>
Pid: 8321, comm: mnt_monitor Tainted: G        W  -- ------------    2.6.32-754.6.3.el6.x86_64 
RIP: 0010:[<ffffffffa0310a0f>]  [<ffffffffa0310a0f>] redirfs_detach_data_file+0x4f/0xd0 [redirfs]

Or

Raw

BUG: unable to handle kernel paging request at 00000001635e1065
IP: [<ffffffffa0312a0f>] redirfs_detach_data_file+0x4f/0xd0 [redirfs]

决议

• Panic has occurred in 'redirfs' module which is a third party code. For further investigation please contact vendor who has provided this module.

根源

• Kernel has panicked in third party redirfs module.

诊断步骤

System status when it panicked:

Raw

      KERNEL: /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-504.3.3.el6.x86_64/vmlinux
    DUMPFILE: /cores/retrace/tasks/299866732/crash/vmcore  [PARTIAL DUMP]
        CPUS: 4
        DATE: Fri May  8 09:24:41 2015
      UPTIME: 11:45:43
LOAD AVERAGE: 1.87, 1.41, 0.91
       TASKS: 1058
    NODENAME: localhost
     RELEASE: 2.6.32-504.3.3.el6.x86_64
     VERSION: #1 SMP Fri Dec 12 16:05:43 EST 2014
     MACHINE: x86_64  (2700 Mhz)
      MEMORY: 16 GB
       PANIC: "kernel BUG at /build/DEEPSECURITY_DSA/9.5_SP1/rhel6_64/en/Source/src/output/RedHat_EL6.x86_64/vfs-filter/2.6.32-431.el6.x86_64/3rd_party/redirfs-0.10/rfs_inode.c:114!"
         PID: 1918
     COMMAND: "glusterfs"
        TASK: ffff8804368c9540  [THREAD_INFO: ffff88043638e000]
         CPU: 2
       STATE: TASK_RUNNING (PANIC)

2) There are 3 third party modules inserted in this system:

Raw

mod: cannot find or load object file for dsa_filter module
mod: cannot find or load object file for redirfs module
mod: cannot find or load object file for gsch module

crash> mod -t
NAME        TAINTS
dsa_filter  P(U)
redirfs     (U)
gsch        (U)

3) Actual panic messages:

Raw

------------[ cut here ]------------
kernel BUG at /build/DEEPSECURITY_DSA/9.5_SP1/rhel6_64/en/Source/src/output/RedHat_EL6.x86_64/vfs-filter/2.6.32-431.el6.x86_64/3rd_party/redirfs-0.10/rfs_inode.c:114!   <<<<<--<<<<<--- Note this, it is coming from 'redirfs' code, which is third party
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/online
CPU 2 
Modules linked in: gsch(U) redirfs(U) fuse nfs lockd fscache auth_rpcgss nfs_acl sunrpc nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ip6table_filter ip6_tables iptable_filter ip_tables ipv6 dsa_filter(P)(U) ppdev parport_pc parport microcode e1000 vmware_balloon i2c_piix4 i2c_core sg shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom mptspi mptscsih mptbase scsi_transport_spi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 1918, comm: glusterfs Tainted: P           ---------------    2.6.32-504.3.3.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
RIP: 0010:[<ffffffffa03c0cb9>]  [<ffffffffa03c0cb9>] rfs_inode_put+0x69/0x70 [redirfs]
RSP: 0018:ffff88043638f4c8  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8803494fa528 RCX: ffff8803ae051518
RDX: 0000000000000034 RSI: ffff8803494fa528 RDI: ffff8803494fa528
RBP: ffff88043638f4d8 R08: ffff88043804d1d0 R09: dead000000200200
R10: ffff8802a6506a68 R11: ffff8803bcba5138 R12: ffff8802c5a895d0
R13: ffff8802c5a89670 R14: ffff8804358fef80 R15: ffff88043638f538
FS:  00007f99cfde7700(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fc5606e2000 CR3: 00000004375cd000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process glusterfs (pid: 1918, threadinfo ffff88043638e000, task ffff8804368c9540)
Stack:
 ffff8802c5a895d0 ffff8803494fa528 ffff88043638f508 ffffffffa03bee68
<d> ffff88043638f508 ffff8803ae0514e8 ffff88043638f578 ffff8803ae0514e8
<d> ffff88043638f528 ffffffffa03c25b4 ffff8802c5a895d0 ffff88043638f578
Call Trace:
 [<ffffffffa03bee68>] rfs_dentry_rem_rinode+0x68/0x80 [redirfs]
 [<ffffffffa03c25b4>] rfs_dcache_rinode_del+0x14/0xa0 [redirfs]
 [<ffffffffa03bf8d9>] rfs_d_iput+0x129/0x240 [redirfs]
 [<ffffffff811a7bec>] dentry_iput+0x7c/0x100
 [<ffffffff811a7d61>] d_kill+0x31/0x60
 [<ffffffff811a80f6>] __shrink_dcache_sb+0x366/0x3c0
 [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff811a8279>] shrink_dcache_memory+0x129/0x1e0
 [<ffffffff8113d4ba>] shrink_slab+0x12a/0x1a0
 [<ffffffff81140096>] zone_reclaim+0x356/0x570
 [<ffffffff811321dc>] get_page_from_freelist+0x6ac/0x870
 [<ffffffff81133d53>] __alloc_pages_nodemask+0x113/0x8d0
 [<ffffffff814b1fc4>] ? tcp_transmit_skb+0x4b4/0x8b0
 [<ffffffff811750ad>] ? __kmalloc_node+0x4d/0x60
 [<ffffffff8145066a>] ? __alloc_skb+0x7a/0x180
 [<ffffffff8116c69a>] alloc_pages_current+0xaa/0x110
 [<ffffffff814a51a7>] tcp_sendmsg+0x677/0xa20
 [<ffffffff8144a7ab>] sock_aio_write+0x19b/0x1c0
 [<ffffffffa0301ac0>] ? fuse_dev_write+0x0/0x70 [fuse]
 [<ffffffff8144a610>] ? sock_aio_write+0x0/0x1c0
 [<ffffffff8118dc1b>] do_sync_readv_writev+0xfb/0x140
 [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8122ef00>] ? cap_file_permission+0x0/0x10
 [<ffffffff8122d4d6>] ? security_file_permission+0x16/0x20
 [<ffffffff8118ec76>] do_readv_writev+0xd6/0x1f0
 [<ffffffff8118edd6>] vfs_writev+0x46/0x60
 [<ffffffff8118ef01>] sys_writev+0x51/0xb0
 [<ffffffff811a995a>] ? dput+0x9a/0x150
 [<ffffffffa048469a>] gsch_writev_hook_fn+0x9a/0x150 [gsch]
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: e8 4d bc ff ff 48 8d 7b 10 e8 74 3b 00 00 48 8b 3d 95 79 00 00 48 89 de e8 05 46 db e0 f0 ff 0d 7e 79 00 00 48 83 c4 08 5b c9 c3 <0f> 0b eb fe 0f 1f 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 00 00 
RIP  [<ffffffffa03c0cb9>] rfs_inode_put+0x69/0x70 [redirfs]
 RSP <ffff88043638f4c8>

Second vmcore:

1) System status when it panicked:

Raw

crash> sys
      KERNEL: /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-504.3.3.el6.x86_64/vmlinux
    DUMPFILE: /cores/retrace/tasks/534341116/crash/vmcore  [PARTIAL DUMP]
        CPUS: 4
        DATE: Fri May  8 22:41:57 2015
      UPTIME: 13:15:07
LOAD AVERAGE: 0.25, 0.21, 0.60
       TASKS: 437
    NODENAME: localhost
     RELEASE: 2.6.32-504.3.3.el6.x86_64
     VERSION: #1 SMP Fri Dec 12 16:05:43 EST 2014
     MACHINE: x86_64  (2700 Mhz)
      MEMORY: 16 GB
       PANIC: "kernel BUG at /build/DEEPSECURITY_DSA/9.5_SP1/rhel6_64/en/Source/src/output/RedHat_EL6.x86_64/vfs-filter/2.6.32-431.el6.x86_64/3rd_party/redirfs-0.10/rfs_inode.c:114!"

2) Third party modules:

Raw

crash>    mod -t
NAME        TAINTS
dsa_filter  P(U)
redirfs     (U)
gsch        (U)

3) Log messages:

Raw

[22782(glusterfs)]: gsch_mount_hook_fn(10.0.0.2:/FEPSPublic,/fepspublic,fuse.glusterfs,0,000000000062f1c0) done
gsch_flt_add_mnt(/opt/tmp @ 10.153.242.59:/SOAPEE[65735546(fuse.glusterfs)]) done: 0
[22819(glusterfs)]: gsch_mount_hook_fn(10.0.0.2:/SOAPEE,/opt/tmp,fuse.glusterfs,0,000000000062f180) done
gsch_flt_add_mnt(/ffm @ 10.153.242.59:/xfervol[65735546(fuse.glusterfs)]) done: 0
[22856(glusterfs)]: gsch_mount_hook_fn(10.0.0.2:/xfervol,/ffm,fuse.glusterfs,0,000000000062f180) done
shrink_slab: shrink_dcache_memory+0x0/0x1e0 negative objects to delete nr=-74363813888
shrink_slab: shrink_dcache_memory+0x0/0x1e0 negative objects to delete nr=-37181906944
shrink_slab: shrink_dcache_memory+0x0/0x1e0 negative objects to delete nr=-74363813888
shrink_slab: shrink_dcache_memory+0x0/0x1e0 negative objects to delete nr=-75525748480
------------[ cut here ]------------
kernel BUG at /build/DEEPSECURITY_DSA/9.5_SP1/rhel6_64/en/Source/src/output/RedHat_EL6.x86_64/vfs-filter/2.6.32-431.el6.x86_64/3rd_party/redirfs-0.10/rfs_inode.c:114!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/online
CPU 2 
Modules linked in: gsch(U) redirfs(U) fuse nfs lockd fscache auth_rpcgss nfs_acl sunrpc nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ip6table_filter ip6_tables iptable_filter ip_tables ipv6 dsa_filter(P)(U) ppdev parport_pc parport microcode e1000 vmware_balloon sg i2c_piix4 i2c_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom mptspi mptscsih mptbase scsi_transport_spi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]

Pid: 21985, comm: perl Tainted: P           ---------------    2.6.32-504.3.3.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
RIP: 0010:[<ffffffffa0305cb9>]  [<ffffffffa0305cb9>] rfs_inode_put+0x69/0x70 [redirfs]
RSP: 0018:ffff8801d9b97508  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff880372d24ea8 RCX: ffff880431d9b900
RDX: 000000000000002e RSI: ffff880372d24ea8 RDI: ffff880372d24ea8
RBP: ffff8801d9b97518 R08: ffff8803975c2610 R09: dead000000200200
R10: 0000000000000001 R11: 0000000000000000 R12: ffff8801df678db0
R13: ffff8801df678e50 R14: ffff880437182f40 R15: ffff8801d9b97578
FS:  00007fc1a4037700(0000) GS:ffff880028300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fc15811e000 CR3: 000000040510b000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process perl (pid: 21985, threadinfo ffff8801d9b96000, task ffff88031d74b500)
Stack:
 ffff8801df678db0 ffff880372d24ea8 ffff8801d9b97548 ffffffffa0303e68
<d> ffff8801d9b97548 ffff880431d9b8d0 ffff8801d9b975b8 ffff880431d9b8d0
<d> ffff8801d9b97568 ffffffffa03075b4 ffff8801df678db0 ffff8801d9b975b8
Call Trace:
 [<ffffffffa0303e68>] rfs_dentry_rem_rinode+0x68/0x80 [redirfs]
 [<ffffffffa03075b4>] rfs_dcache_rinode_del+0x14/0xa0 [redirfs]
 [<ffffffffa03048d9>] rfs_d_iput+0x129/0x240 [redirfs]
 [<ffffffff811a7bec>] dentry_iput+0x7c/0x100
 [<ffffffff811a7d61>] d_kill+0x31/0x60
 [<ffffffff811a80f6>] __shrink_dcache_sb+0x366/0x3c0
 [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff8100bb8e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff811a8279>] shrink_dcache_memory+0x129/0x1e0
 [<ffffffff8113d4ba>] shrink_slab+0x12a/0x1a0
 [<ffffffff81140096>] zone_reclaim+0x356/0x570
 [<ffffffff811321dc>] get_page_from_freelist+0x6ac/0x870
 [<ffffffff812717bc>] ? blk_queue_bio+0x11c/0x610
 [<ffffffff81133d53>] __alloc_pages_nodemask+0x113/0x8d0
 [<ffffffff8116c69a>] alloc_pages_current+0xaa/0x110
 [<ffffffff811246f7>] __page_cache_alloc+0x87/0x90
 [<ffffffff81139ddb>] __do_page_cache_readahead+0xdb/0x210
 [<ffffffff81139f31>] ra_submit+0x21/0x30
 [<ffffffff8113a2bd>] ondemand_readahead+0x11d/0x250
 [<ffffffff8113a480>] page_cache_async_readahead+0x90/0xc0
 [<ffffffff81125fb3>] generic_file_aio_read+0x503/0x700
 [<ffffffff8118de9a>] do_sync_read+0xfa/0x140
 [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8118e0d2>] ? vfs_write+0x132/0x1a0
 [<ffffffff8122d4d6>] ? security_file_permission+0x16/0x20
 [<ffffffff8118e855>] vfs_read+0xb5/0x1a0
 [<ffffffff8118e991>] sys_read+0x51/0x90
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: e8 4d bc ff ff 48 8d 7b 10 e8 74 3b 00 00 48 8b 3d 95 79 00 00 48 89 de e8 05 f6 e6 e0 f0 ff 0d 7e 79 00 00 48 83 c4 08 5b c9 c3 <0f> 0b eb fe 0f 1f 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 00 00 
RIP  [<ffffffffa0305cb9>] rfs_inode_put+0x69/0x70 [redirfs]
 RSP <ffff8801d9b97508>

One more example of crash:

Raw

crash> sys
      KERNEL: /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-754.6.3.el6.x86_64/vmlinux
    DUMPFILE: /cores/retrace/tasks/122044735/crash/vmcore  [PARTIAL DUMP]
        CPUS: 6
        DATE: Sat Jun  1 14:01:46 2019
      UPTIME: 175 days, 13:00:51
LOAD AVERAGE: 3.30, 3.34, 3.29
       TASKS: 1440
    NODENAME: localhost
     RELEASE: 2.6.32-754.6.3.el6.x86_64
     VERSION: #1 SMP Tue Sep 18 10:29:08 EDT 2018
     MACHINE: x86_64  (2400 Mhz)
      MEMORY: 40 GB
       PANIC: "general protection fault: 0000 [#1] SMP "
crash> mod -t
NAME            TAINTS
redirfs         (U)
ampavflt        (U)
ampnetworkflow  (U)
vmci            (U)
vsock           (U) 

Backtrace:

Raw

crash> bt
PID: 8321   TASK: ffff880242835520  CPU: 5   COMMAND: "mnt_monitor"
 #0 [ffff880a2a1df8e0] machine_kexec at ffffffff81040f1b
 #1 [ffff880a2a1df940] crash_kexec at ffffffff810d6722
 #2 [ffff880a2a1dfa10] oops_end at ffffffff8155e2e0
 #3 [ffff880a2a1dfa40] die at ffffffff8100f95b
 #4 [ffff880a2a1dfa70] do_general_protection at ffffffff8155ddc2
 #5 [ffff880a2a1dfaa0] general_protection at ffffffff8155d235
    [exception RIP: redirfs_detach_data_file+79]
    RIP: ffffffffa0310a0f  RSP: ffff880a2a1dfb58  RFLAGS: 00010206
    RAX: 4c4a45bc7b1016ee  RBX: ffff880a2ab2f000  RCX: ffffffffa02601c0
    RDX: ffff880141182f80  RSI: ffff880242b468c0  RDI: ffff880a2ab2f000
    RBP: ffff880a2a1dfb88   R8: ffff880a2a1dfc38   R9: 0000000000000000
    R10: ffff88000bbcef20  R11: 0000000000000020  R12: ffff880a2ab2f000
    R13: ffff88020ac69538  R14: ffff88020acd8cc0  R15: ffff88020ac69528
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #6 [ffff880a2a1dfb90] rfs_dentry_rem_data at ffffffffa030c2ec [redirfs]
 #7 [ffff880a2a1dfbe0] rfs_dcache_rem at ffffffffa030ee23 [redirfs]
 #8 [ffff880a2a1dfc10] rfs_dcache_walk at ffffffffa030f3eb [redirfs]
 #9 [ffff880a2a1dfc90] rfs_info_rem at ffffffffa030a521 [redirfs]
#10 [ffff880a2a1dfcc0] rfs_info_rem_include at ffffffffa030ab3f [redirfs]
#11 [ffff880a2a1dfd10] rfs_root_rem_include at ffffffffa0309bc2 [redirfs]
#12 [ffff880a2a1dfd50] redirfs_rem_path at ffffffffa0308fb7 [redirfs]
#13 [ffff880a2a1dfd90] redirfs_rem_paths at ffffffffa0309148 [redirfs]
#14 [ffff880a2a1dfdd0] rfs_flt_paths_store at ffffffffa03116ea [redirfs]
#15 [ffff880a2a1dfe20] rfs_flt_store at ffffffffa0311920 [redirfs]
#16 [ffff880a2a1dfe60] sysfs_write_file at ffffffff8121fb8c
#17 [ffff880a2a1dfec0] vfs_write at ffffffff811a155a
#18 [ffff880a2a1dff00] sys_write at ffffffff811a2091
#19 [ffff880a2a1dff50] system_call_fastpath at ffffffff81564357

 

转载至https://access.redhat.com/solutions/1443703