Problem Description
A JDBC connection which is used by an application or by WebLogic Server itself will block one WebLogic Server execute thread for the complete duration of the calls that are made via this connection. The JVM will ensure that the CPU is given to runnable threads by its thread scheduling mechanism, while the thread that blocks on a SQL query needs to wait. However, the thread occupied by the JDBC call will be reserved and used for the application until the call returns from the SQL query.
Even a transaction timeout will not kill or timeout any action that is done by the resources that are enlisted in this transaction. The actions will run as long as they take, without interruption. A transaction timeout will set a flag on the transaction that will mark it as rollback only, so that any subsequent request to commit this transaction will fail with aTimedOutException or RollbackException. However, as mentioned above, the long running JDBC calls can lead to blocked WebLogic Server execute threads, which can finally lead to a hanging instance, if all threads are blocked and no execute thread remains available for handling incoming requests.
More recent WebLogic Server versions have a health check functionality that regularly checks if a thread does not react for a certain period of time (the default is 600 seconds). If this happens, an error message is printed to your log file similar to following:
The following are some different possible reasons that can cause JDBC calls to lead to a hanging WebLogic Server instance:
- Use of DriverManager.getConnection() in your JDBC code.
- SQL Queries issued to the database take unexpectedly long time to return.
- Database for which the JDBC connection pool is configured hangs and does not return from calls in a timely manner.
- A slow or overloaded network causes database calls to slow down or hang.
- A deadlock causes all execute threads to hang and wait forever.
- RefreshMinutes or TestFrequencySeconds property in the JDBC connection pool causes hang periods in WebLogic Server.
- JDBC connection pool shrinking and re-creation of database connections causes long response times.