I'm developing a web cralwer with htmlunit and I have added all required timeout but I notice that the app hangs when the server of some website been crawled is not responding at when I use the Java VisualVM to do a thread dump: java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.net.SocksSocketImpl.readSocksReply(SocksSocketImpl.java:88) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:429) at java.net.Socket.connect(Socket.java:525) at com.gargoylesoftware.htmlunit.SocksSocketFactory.connectSocket(SocksSocketFactory.java:89) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149) at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:573) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:776) at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:152) at app.plugin.core.net.QHttpWebConnection.getResponse(QHttpWebConnection.java:30) at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1439) at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1358) at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:307) at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:373) at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:358)
This is really frustrating since I have no control of those servers. This issue is seriously affecting the performance of my application.
Question: How can I solve this issue?
Is there a way to get a list of socket connection opened by a Java app and use that to terminate the socket, like simluate that the server closed the connection?