记一次生产数据库系统内存使用过高的案例

墨墨导读：根据监控平台信息，发现数据库平台节点2内存使用率过高，达到98%。通过查询占用内存较高的进程、检查TFA状态、同步TFA配置等方式，使得系统恢复正常运作。

概述

根据监控平台信息，发现某数据库平台节点2内存使用率过高，内存使用率达到98%。

1. 查询占用内存较高的进程

grid     280483 183124  0 18:21 ?        00:00:00 [asmcmd daemon]
grid     280493 155171  0 18:21 ?        00:00:00 [asmcmd daemon]
grid     280497 104733  0 18:21 ?        00:00:00 [asmcmd daemon]
grid     280499 187375  0 18:21 ?        00:00:00 [asmcmd daemon]
grid     280533 239249  0 18:21 ?        00:00:00 [asmcmd daemon]
grid     280534 157752  0 18:21 ?        00:00:00 [asmcmd daemon]
grid     280536 281960  0 18:21 ?        00:00:00 [asmcmd daemon]
grid     280545  69656  0 18:21 ?        00:00:00 [asmcmd daemon]
grid     280552 128541  0 18:21 ?        00:00:00 [asmcmd daemon]
grid     280553  63409  0 18:21 ?        00:00:00 [asmcmd daemon]
grid     280558 108705  0 18:21 ?        00:00:00 [asmcmd daemon]
grid     280575 194378  0 18:21 ?        00:00:00 [asmcmd daemon]

内存使用率暂用最高的为asmcmd daemon，这个进程究竟在做什么导致消耗这么高的内存呢？

记下来跟踪一下该进程过程。

wait4(163639, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 163639
open("/tmp/clsecho_stderr_file.txt", O_RDONLY) = 4
ioctl(4, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7ffc8333d180) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR)                   = 0
fstat(4, {st_mode=S_IFREG|0644, st_size=285, ...}) = 0
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
read(4, "Can't open '/oracle/app/12.2.0/g"..., 8192) = 285
stat("/oracle/app/12.2.0/grid/bin/clsecho", {st_mode=S_IFREG|0755, st_size=11405, ...}) = 0
geteuid()                               = 1001
geteuid()                               = 1001
getegid()                               = 501
lseek(4, 99, SEEK_SET)                  = 99
lseek(4, 0, SEEK_CUR)                   = 99
pipe([6, 7])                            = 0
pipe([8, 9])                            = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f059005ea10) = 164739
close(9)                                = 0
close(7)                                = 0
read(8, "", 4)                          = 0
close(8)                                = 0
ioctl(6, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7ffc8333d0f0) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(6, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
fstat(6, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
brk(0)                                  = 0x81493d000
brk(0x81495e000)                        = 0x81495e000
read(6, "20-Jul-20 18:14 ASMCMD Backgroun"..., 8192) = 102
read(6, "", 8192)                       = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=164739, si_status=0, si_utime=0, si_stime=195} ---
fstat(6, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
close(6)                                = 0
brk(0)                                  = 0x81495e000
brk(0)                                  = 0x81495e000
brk(0x81495c000)                        = 0x81495c000
brk(0)                                  = 0x81495c000
wait4(164739, [{WIFEXITED