今天早上登陆服务器时反应很慢,进去后投票一看,我擦CPU全站满了如下图:
oracle 1158 1.5 0.1 23004 7780 ? R Oct30 22:11 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
oracle 1212 2.7 0.1 23004 7780 ? R Oct27 148:01 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
oracle 1282 3.1 0.1 23008 7784 ? R Oct27 192:43 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
oracle 1367 2.3 0.1 23004 4092 ? R Oct28 110:51 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
oracle 1379 1.6 0.1 23008 7780 ? R 00:12 9:37 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
oracle 1444 2.1 0.1 23008 4092 ? R Oct28 82:42 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
oracle 1465 4.1 0.1 23008 4092 ? R Oct26 286:46 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
oracle 1608 1.8 0.1 23004 7780 ? R Oct29 40:22 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
oracle 1663 1.9 0.1 23004 7780 ? R Oct29 59:45 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
oracle 1789 2.5 0.1 23012 7784 ? R Oct27 140:23 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
oracle 1941 1.6 0.1 23008 7780 ? R 00:27 9:22 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
oracle 1960 2.3 0.1 23004 4092 ? R Oct28 107:43 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl
oracle 2030 2.1 0.1 23008 7780 ? R Oct28 81:42 /home/database/oracle/product/10.2.0/db_1/perl/bin/perl /home/database/oracle/product/10.2.0/db_1/sysman/admin/scripts/db/dbresp.pl wc了一下居然有520个进程,我擦 ,而且应用还停着,关闭数据库这些pl进程仍然存在,最后直接批量kill掉,恢复正常
ps -aux|grep product/10.2.0/db_1/perl/bin/perl|grep -v grep|cut -c 9-15|xargs kill -9
分析问题:
原因:
根据metalink Doc ID: 764140.1描述:The Response metric is making a timed out then the Agent starts other process to take the Response metric. The process to kill the PID taking the Response metric is failing increasing the process running dbresp.pl。
解决方法:
同样根据根据metalink Doc ID: 764140.1:
To implement the solution, please execute the following steps:
1. Stop DBConsole
emctl stop dbconsole
2. Kill any running process.
ps -ef | grep /opt/app/oracle/_
Kill any returned process.
ps -ef |grep '问题进程' |cut -c 10-15 |xargs kill -9
3. Follow fix
Note.361612.1 Ext/Mod Problem Performance Agent High CPU Consumption Gen
4. Start DB Console emctl start dbconsole