oracle报大量的连接超时,11gR2 RAC警告日志大量报监听连接超时TNS-12535

操作系统版本：AIX 6100-08-03-1339

数据库版本：11.2.0.3.7

警告日志报错：

Fatal NI connect error 12170.

VERSION INFORMATION:

TNS for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production

TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production

Oracle Bequeath NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production

Time: 10-SEP-2013 17:04:27

Tracing not turned on.

Tns error struct:

ns main err code: 12535

TNS-12535: TNS:operation timed out

ns secondary err code: 12606

nt main err code: 0

nt secondary err code: 0

nt OS err code: 0

Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=136.24.2.26)(PORT=52128))

WARNING: inbound connection timed out (ORA-3136)

listener日志部分信息：

10-SEP-2013 16:52:07 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=xjocsdb1)(USER=grid))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=186647296)) * status * 0

10-SEP-2013 16:52:12 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=billdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=136.24.2.26)(PORT=53047)) * establish * billdb * 0

10-SEP-2013 16:52:12 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=billdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=136.24.2.26)(PORT=53051)) * establish * billdb * 0

10-SEP-2013 16:52:13 * service_update * billdb1 * 0

Tue Sep 10 16:52:22 2013

10-SEP-2013 16:52:22 * service_update * billdb1 * 0

Tue Sep 10 16:52:33 2013

10-SEP-2013 16:52:33 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=billdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=136.24.2.26)(PORT=53061)) * establish * billdb * 0

10-SEP-2013 16:52:34 * service_update * billdb1 * 0

10-SEP-2013 16:52:41 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=billdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=136.24.2.26)(PORT=53068)) * establish * billdb * 0

Tue Sep 10 16:52:43 2013

10-SEP-2013 16:52:43 * service_update * billdb1 * 0

10-SEP-2013 16:52:52 * service_update * billdb1 * 0

Tue Sep 10 16:53:07 2013

10-SEP-2013 16:53:07 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=xjocsdb1)(USER=grid))(COMMAND=status)(ARGUMENTS=64)(SERVICE=LISTENER)(VERSION=186647296)) * status * 0

10-SEP-2013 16:53:10 * service_update * billdb1 * 0

10-SEP-2013 16:53:12 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=billdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=136.24.2.26)(PORT=53084)) * establish * billdb * 0

10-SEP-2013 16:53:13 * service_update * billdb1 * 0

Tue Sep 10 16:53:22 2013

10-SEP-2013 16:53:22 * service_update * billdb1 * 0

Tue Sep 10 16:53:33 2013

10-SEP-2013 16:53:33 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=billdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=136.24.2.26)(PORT=53094)) * establish * billdb * 0

10-SEP-2013 16:53:34 * service_update * billdb1 * 0

10-SEP-2013 16:53:42 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))(SERVICE_NAME=billdb)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=oracle))) * (ADDRESS=(PROTOCOL=tcp)(HOST=136.24.2.26)(PORT=53101)) * establish * billdb * 0

Tue Sep 10 16:53:43 2013

10-SEP-2013 16:53:43 * service_update * billdb1 * 0

10-SEP-2013 16:53:45 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=serv_taf_billdb)(CID=(PROGRAM=AM_Server_A11)(HOST=xjocsapp1)(USER=ocsext))) * (ADDRESS=(PROTOCOL=tcp)(HOST=136.24.2.37)(PORT=57554)) * establish * serv_taf_billdb * 0

10-SEP-2013 16:53:45 * (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=serv_taf_billdb)(CID=(PROGRAM=AM_Server_A11)(HOST=xjocsapp1)(USER=ocsext))) * (ADDRESS=(PROTOCOL=tcp)(HOST=136.24.2.37)(PORT=57562)) * establish * serv_taf_billdb * 0

sqlnet已经设置了超时时间，但不管用

SQLNET.INBOUND_CONNECT_TIMEOUT=180

监听日志也设置了INBOUND_CONNECT_TIMEOUT_LISTENER=120

但警告日志依然报错。

]$ netstat -an|grep 1521|wc -l

82

]$ ps -ef|grep LOCAL=NO|wc -l

394

连接数还在继续增长，估计是超时导致大量重连接所致。

求助一下各位