重点排查一下异常期间的ps 数据,排除S(sleep)状态进程,可以看到大量的IO相关D进程
[oracle@hn1dbn01 tmp]$ cat /tmp/ps1024.txt | awk '{print $1,$9,$10,$13,$14}' | sort | uniq -c454 oracle filp_o D oracleemktdb1 (LOCAL=NO)
3 oracle get_wr D oracleemktdb1 (LOCAL=NO)
35 oracle lookup D oracleemktdb1 (LOCAL=NO)
1 oracle lookup D ora_m001_emktdb1
1 oracle - R ps -aeo
16 oracle sync_b D oracleemktdb1 (LOCAL=NO)
1 root ? R /bin/netstat -ap
2 root ? R /bin/ps -p
[oracle@hn1dbn01 tmp]$
[oracle@hn1dbn01 tmp]$ cat /tmp/ps1024.txt | grep '10:00' | awk '{print $1,$9,$10,$11,$13,$14}' | sort | uniq -c112 oracle filp_o D 10:00:14 oracleemktdb1 (LOCAL=NO)
115 oracle filp_o D 10:00:15 oracleemktdb1 (LOCAL=NO)
75 oracle filp_o D 10:00:16 oracleemktdb1 (LOCAL=NO)9 oracle filp_o D 10:00:17 oracleemktdb1 (LOCAL=NO)1 oracle filp_o D 10:00:36 oracleemktdb1 (LOCAL=NO)6 oracle lookup D 10:00:12 oracleemktdb1 (LOCAL=NO)38 oracle lookup D 10:00:13 oracleemktdb1 (LOCAL=NO)33 oracle lookup D 10:00:14 oracleemktdb1 (LOCAL=NO)33 oracle lookup D 10:00:15 oracleemktdb1 (LOCAL=NO)47 oracle lookup D 10:00:16 oracleemktdb1 (LOCAL=NO)17 oracle lookup D 10:00:17 oracleemktdb1 (LOCAL=NO)5 oracle lookup D 10:00:18 oracleemktdb1 (LOCAL=NO)
排查ash信息再次确认数据库负载较低,io响应也在毫秒级,无明显异常
使用strace跟踪测试建立连接期间,发现监听派生 LOCAL=NO进程后会读取大量的ORACLE_HOME本地lib库,并进一步排查/u01文件系统状态
[oracle@hn1dbn01 tmp]$ cat /tmp/stsqlplus.txt | grep '12.2.0/db'126319 open("/u01/app/oracle/product/12.2.0/db/lib/libsqlplus.so", O_RDONLY) = 3126319 open("/u01/app/oracle/product/12.2.0/db/lib/libclntsh.so.12.1", O_RDONLY) = 3126319 open("/u01/app/oracle/product/12.2.0/db/lib/libclntshcore.so.12.1", O_RDONLY) = 3126319 open("/u01/app/oracle/product/12.2.0/db/lib/libmql1.so", O_RDONLY) = 3126319 open("/u01/app/oracle/product/12.2.0/db/lib/libipc1.so", O_RDONLY) = 3126319 open("/u01/app/oracle/product/12.2.0/db/lib/libnnz12.so", O_RDONLY) = 3
[oracle@hn1dbn01 tmp]$ lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTsda 8:0 0 837.3G 0 disksda1 8:1 0 1G 0 part /bootsda2 8:2 0 100G 0 part /sda3 8:3 0 20G 0 part [SWAP]sda4 8:4 0 1K 0 partsda5 8:5 0 20G 0 part /optsda6 8:6 0 20G 0 part /usrsda7 8:7 0 20G 0 part /var/logsda8 8:8 0 200G 0 part /u01
sdb 8:16 0 1.5T 0 disk
.....sdo 8:224 0 30G 0 disksdp 8:240 0 300G 0 disksdq 65:0 0 300G 0 disksdr 65:16 0 2T 0 diskasm!oradata-127 251:65025 0 2T 0 disk
zzz ***Wed Aug 1410:00:11CST 2019avg-cpu: %user %nice %system %iowait %steal %idle11.35 0.00 2.83 3.21 0.00 82.61
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 169.00 5.00 642.00 8.00 29000.00 52.00 89.39 10.19 35.69 2.54 100.00