我们在Websphere Application Server 7.0.0上部署了一些CXF Web服务。 它是带有负载均衡器的多服务器环境。
我们有一个多线程Web服务客户端(用作缓存工具),该客户端调用Web服务和来自调用方的服务请求。 所以本质上,请求流是这样的-
调用方-> CXF WebService客户端(多线程)-> F5负载平衡器-> Websphere AS
一切正常,但是有时我们在WebService客户端上看到“ java.net.SocketTimeoutException:读取超时”。 在一个Websphere AS(正在处理此传入请求)上,我们看到“ java.io.IOException:SRVE0080E:无效的内容长度”。 我们无法在测试环境中重现此问题,并且会在客户的生产环境中随机发生。 发生的频率不是很高,我们验证了负载也没有那么多。 请求不包含太多数据,它只是客户端从服务器请求的对象的“ id”。 由于服务器从未收到完整的请求数据,因此永远不会调用webservice方法。
客户端的完整堆栈跟踪-
[pool-1-thread-643] WARN phase.PhaseInterceptorChain Interceptor has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message.
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:472)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:302)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:254)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:73)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:123)
at $Proxy55.getObject(Unknown Source)
at com.***.****.***service.***Thread.retrieve(***Thread.java:344)
at com.***.****.***service.***Thread.handleMessage(***Thread.java:232)
at com.***.****.***service.***Thread.run(***Thread.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:2046)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:2022)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1947)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:66)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:632)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
... 13 more
全栈跟踪服务器端-
java.io.IOException: SRVE0080E: Invalid content length
at com.ibm.ws.webcontainer.srt.http.HttpInputStream.finish(HttpInputStream.java:184)
at com.ibm.ws.webcontainer.srt.http.HttpInputStream.close(HttpInputStream.java:532)
at com.ibm.ws.webcontainer.srt.SRTServletRequest.finish(SRTServletRequest.java:2617)
at com.ibm.ws.webcontainer.srt.SRTConnectionContext.finishConnection(SRTConnectionContext.java:80)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1048)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:191)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:516)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:307)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:84)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1818)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1662)
有谁能猜到会发生什么? JVM可能在使Web服务调用的线程处于等待或挂起状态,然后它才能完成请求,之后套接字超时,因为JVM在1分钟内没有再次重新安排该线程的时间(我们已将套接字超时设置为1分钟)? 负载均衡器是否可能无法将整个请求数据转发到服务器?
在此先感谢您的帮助。