Red Hat Enterprise Linux 7 server reboots when Trend Micro DS agent is updated

环境

• Red Hat Enterprise Linux 7
• Trend Micro Deep Security Agent 9.6.2-7888.el7 (Kernel modules gsch and redirfs)
• Imperva Agent 11.5.0.5030 (kernel module krg_11_5_0_5030_imRH7K1smp64)

问题

• What is the Red Hat Enterprise Linux 7 server rebooting after applying Trend Micro DS agent updates ?
• While applying Trend Micro DS agent update, server reboots with the following messages

Raw

[726704.698199] gsch_dev_release() doing
[726704.698208] gsch_dev_release() done
[726705.902410] unhooking open NR=2 ... ffffffffa02bc930 <- ffffffff811fd550
[726705.902418] could not restore 'close' system call
[726705.902421] unhooking close NR=3 ... ffffffffa08dc8e0 <- ffffffff811fd5b0
[726705.902423] could not restore 'exit' system call
[726705.902426] unhooking exit NR=60 ... ffffffffa08de010 <- ffffffff8108c4b0
[726705.902428] unhooking getpgid NR=121 ... ffffffffa02ba800 <- ffffffff810a29e0
[726705.902431] unhooking unlink NR=87 ... ffffffffa02bae50 <- ffffffff81210470
[726705.902433] unhooking unlinkat NR=263 ... ffffffffa02bb0f0 <- ffffffff81210430
[726705.902435] could not restore 'write' system call
[726705.902438] unhooking write NR=1 ... ffffffffa08d9920 <- ffffffff811fed30
[726705.902440] unhooking pwrite64 NR=18 ... ffffffffa02bac80 <- ffffffff811feed0
[726705.902442] could not restore 'writev' system call
[726705.902444] unhooking writev NR=20 ... ffffffffa08d9b60 <- ffffffff811ff4e0
[726705.902446] unhooking dup2 NR=33 ... ffffffffa02bb930 <- ffffffff8121d210
[726705.902448] unhooking mount NR=165 ... ffffffffa02bcba0 <- ffffffff81220f60
[726705.902450] unhooking umount NR=166 ... ffffffffa02bc090 <- ffffffff8121fa60
[726705.902452] could not restore 'exit_group' system call
[726705.902454] unhooking exit_group NR=231 ... ffffffffa08de060 <- ffffffff8108c570
[726705.902455] Failed to remove all hooked system calls.
[726705.904328] System may be left in an unstable state.
[726705.905890] Failed to remove hooked execve().
[726705.907429] System may be left in an unstable state.
[726705.908974] gsch_remove_hooks(&gsch_hooks, &orig_hooks) done: -5
[726705.909322] gsch_flt: unloading
[726706.384259] gsch_flt: unloaded
[726706.396213] Failed to remove hooked execve().
[726706.398083] System may be left in an unstable state.
[726706.399650] unregister_chrdev(242) done
[726706.400160] vfree(gsch_cache=ffffc900612d3000)
[726706.400166] gsch: unloading vfs-filter 9.6.2.7976: OK
[374052.252532] BUG: unable to handle kernel paging request at ffffffffa03607a0
[374052.254292] IP: [] 0xffffffffa036079f
[374052.255879] PGD 19bd067 PUD 19be063 PMD fe7744f067 PTE 0
[374052.257454] Oops: 0010 [#1] SMP 
[374052.259005] Modules linked in: ip6table_filter ip6_tables iptable_filter krg_11_5_0_5030_imRH7K1smp64(POE) binfmt_misc redirfs(OE) dsa_filter(POE) bonding iTCO_wdt iTCO_vendor_support dcdbas intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i2c_i801 sb_edac edac_core ipmi_devintf mei_me mei lpc_ich ipmi_si ipmi_msghandler dm_round_robin wmi shpchp acpi_power_meter tpm_crb sg dm_multipath ip_tables xfs sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci qla2xxx bnx2x libata i2c_core mdio scsi_transport_fc ptp megaraid_sas pps_core scsi_tgt
[374052.269563]  libcrc32c fjes dm_mirror dm_region_hash dm_log dm_mod [last unloaded: gsch]
[374052.271540] CPU: 30 PID: 176181 Comm: ds_am.init Tainted: P           OE  ------------   3.10.0-514.2.2.el7.x86_64 #1
[374052.273427] Hardware name: Dell Inc. PowerEdge R930/01FH6X, BIOS 2.3.1 01/09/2017
[374052.275326] task: ffff88a401fd0fb0 ti: ffff88c64e264000 task.ti: ffff88c64e264000
[374052.277291] RIP: 0010:[]  [] 0xffffffffa036079f
[374052.279211] RSP: 0018:ffff88c64e267d58  EFLAGS: 00010286
[374052.281099] RAX: ffffffffa03607a0 RBX: ffff88c64e267e98 RCX: 0000000000000001
[374052.283017] RDX: 0000000000e63260 RSI: 0000000000e6eae0 RDI: 0000000000e640d0
[374052.285054] RBP: ffff88c64e267d60 R08: 0000000000000001 R09: ffff884f69fd8840
[374052.287042] R10: 0000000000000000 R11: ffffea03f646e840 R12: ffffffffa0ca39f8
[374052.288966] R13: 00000000000001d8 R14: ffffffffa0b9fab0 R15: 0000000000000055
[374052.290924] FS:  00007fc9abedc740(0000) GS:ffff887e7e9c0000(0000) knlGS:0000000000000000
[374052.292866] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[374052.294860] CR2: ffffffffa03607a0 CR3: 000000c66c666000 CR4: 00000000003407e0
[374052.296812] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[374052.298727] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[374052.300631] Stack:
[374052.302512]  ffffffffa0c34ed7 ffff88c64e267db8 ffffffffa0b94fc5 ffff88fe0002b034
[374052.304491]  ffff88c64e267f00 0000000100000000 000000005a631119 000000000002b034
[374052.306437]  0000000000000001 ffffffffa0b8c380 000000000002b035 ffffffffa0b9fab0
[374052.308393] Call Trace:
[374052.310370]  [] ? SYS_execve_auxiliary64+0x27/0xc0 [krg_11_5_0_5030_imRH7K1smp64]
[374052.312385]  [] _execve_origcode+0x125/0x310 [krg_11_5_0_5030_imRH7K1smp64]
[374052.314466]  [] ? _fork_postcode+0x360/0x360 [krg_11_5_0_5030_imRH7K1smp64]
[374052.316547]  [] ? _syscall_wrappers_actions_before_calling_orig_execve+0xe20/0xe20 [krg_11_5_0_5030_imRH7K1smp64]
[374052.320575]  [] syscall_wrappers_generic_flow_with_param+0x39a/0x7d0 [krg_11_5_0_5030_imRH7K1smp64]
[374052.322672]  [] ? do_read_fault.isra.42+0xe4/0x130
[374052.324779]  [] syscall_wrappers_generic_flow+0x12/0x60 [krg_11_5_0_5030_imRH7K1smp64]
[374052.326929]  [] SYS_execve_common_wrap+0x6d/0xe0 [krg_11_5_0_5030_imRH7K1smp64]
[374052.329186]  [] SYS_execve_helper64+0xae/0xd0 [krg_11_5_0_5030_imRH7K1smp64]
[374052.331349]  [] stub_execve+0x69/0xa0
[374052.333482] Code:  Bad RIP value.
[374052.335617] RIP  [] 0xffffffffa036079f
[374052.337767]  RSP 
[374052.339980] CR2: ffffffffa03607a0

决议

Contact Trend Micro and Imperva for a workaround or a fix.

根源

During DS agent updates, ds agent service is reloaded. This results in the unloading of gsch kernel driver. gsch kernel drivers seems to set hooks on certain syscall especially on stub. When the module is unloaded, it does not cleanly remove the hooks which causes Imperva kernel module to fail and result in the crash.