1,问题描述
同事说crm登录hang住了,tomcat后台应用报错如下:
2016-06-28 10:30:11,214 [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] WARN [com.mchange.v2.resourcepool.BasicResourcePool] - com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3930f3cc -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: java.sql.SQLException: Io exception: Message file 'oracle.net.mesg.Message' is missing. at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112) at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146) at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387) at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:414) at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165) at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014) at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32) at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547) 2016-06-28 10:30:56,213 [com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0] WARN [com.mchange.v2.resourcepool.BasicResourcePool] - com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7127f4c0 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: java.sql.SQLException: Io exception: Message file 'oracle.net.mesg.Message' is missing. at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112) at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146) at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387) at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:414) at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165) at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014) at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32) at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Jun 28, 2016 10:33:59 AM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run SEVERE: Socket accept failed java.net.SocketException: Too many open files at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:222) at java.lang.Thread.run(Thread.java:745)
|
2,查看数据库能否正常操作
看到有信息java.sql.SQLException: Ioexception: Message file 'oracle.net.mesg.Message' is missing. 先去查看数据库是否正常:通过plsql连接oracle数据库,显示数据库正常:
SQL> create table zz_bak (id number);
Table created
SQL> insert into zz_bak(id) values(1);
1 row inserted
SQL> commit;
Commit complete
SQL> select * from zz_bak;
ID
----------
1
SQL> drop table zz_bak;
Table dropped
SQL>
3,查看是否有数据库锁
-- 用dba权限的用户查看数据库都有哪些锁,没有查询到记录
selectt2.username,t2.sid,t2.serial#,t2.logon_time from v$locked_object t1,v$sessiont2 where t1.session_id=t2.sid order by t2.logon_time;
-- 查看session等待,也没用记录
SELECT/*+ rule */ lpad(' ', decode(l.xidusn, 0, 3, 0)) || l.oracle_usernameUser_name,o.owner,o.object_name,o.object_type,s.sid,s.serial#,p.spid
FROMv$locked_object l, dba_objects o, v$session s, v$process p
WHEREl.object_id = o.object_id
AND l.session_id = s.sid and s.paddr =p.addr
ORDERBY o.object_id, xidusn DESC
4,检查应用程序所在的linux服务器
看到有“java.net.SocketException:Too many open files”信息,查看打开文件数:
[tomcat@crmtestlogs]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 112257
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 65535
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[tomcat@crmtestlogs]$
查看系统句柄文件数,当前系统文件句柄的最大数目,只用于查看,不能设置修改
# cat /proc/sys/fs/file-max
[tomcat@crmtestlogs]$ cat/proc/sys/fs/file-max
1427648
[tomcat@crmtestlogs]$
如果需要查看所有进程的文件打开数,如下图命令lsof |wc –l:
[tomcat@crmtestlogs]$ lsof |wc -l
1508
[tomcat@crmtestlogs]$
分析得出:文件句柄数量有些多,但是打开文件数还好不算太多
文件句柄数有些多,所以采用解决办法是:重启tomcat应用服务器,释放打开的文件句柄,之后观察5分钟,再也不会报错了。