高并发压测服务端报异常org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe

最新推荐文章于 2024-08-21 09:30:00 发布

工作QQ感冒的石头

最新推荐文章于 2024-08-21 09:30:00 发布

阅读量5.9k

点赞数 1

分类专栏：高并发 java 文章标签： java apache 开发语言

本文链接：https://blog.csdn.net/qq_39291929/article/details/108754188

版权

java 同时被 2 个专栏收录

99 篇文章 3 订阅

订阅专栏

高并发

3 篇文章 0 订阅

订阅专栏

异常信息如下：

020-09-22 18:21:08 ERROR http-nio-8095-exec-16916 io.renren.common.exception.RRExceptionHandler java.io.IOException: Broken pipe
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
   at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351)
   at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:776)
   at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:681)
   at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:386)
   at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:364)
   at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
   at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2137)
   at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1150)
   at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:923)
   at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:287)
   at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:104)
   at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:295)
   at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:181)
   at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:82)
   at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:124)
   at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888)
   at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
   at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
   at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
   at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
   at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
   at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
   at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at io.renren.common.component.filter.MdcFilter.doFilter(MdcFilter.java:26)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at io.renren.common.xss.XssFilter.doFilter(XssFilter.java:45)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
   at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
   at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
   at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
   at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
   at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
   at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
   at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
   at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
   at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
   at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
   at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
   at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
   at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
   at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
   at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
   at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
   at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
   at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
   at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1598)
   at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
   at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Broken pipe
   at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
   at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
   at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
   at sun.nio.ch.IOUtil.write(IOUtil.java:65)
   at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
   at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:138)
   at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
   at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152)
   at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1257)
   at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:741)
   at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:561)
   at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:505)
   at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:538)
   at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84)
   at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:190)
   at org.apache.coyote.Response.doWrite(Response.java:601)
   at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:339)
   ... 83 common frames omitted

看以上报错信息，是tomcat的connector在执行写操作的时候发生了Broken pipe异常，我们分析一下TCPIP
netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"\t",state[key]}'

[root@nice-product02 ~]# netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"\t",state[key]}'
CLOSE_WAIT    2000
ESTABLISHED    459
FIN_WAIT2    27
TIME_WAIT    33

CLOSE_WAIT 状态的连接竟然有2000个，这太不正常了，这说明是客户端先关闭了连接，服务器端没有执行关闭连接的操作，导致服务器端一直维持在CLOSE_WAIT的状态，如果不对操作系统的keepalive做优化，这个状态默认会维持两个小时，查看了下系统的设置：
sysctl -a |grep keepalive

net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200

以上说明TCP发送keepalive消息的频度2个小时

结论：原来这个异常是客户端读取超时关闭了连接,这时候服务器端再向客户端已经断开的连接写数据时就发生了broken pipe异常！
解释：出现大量close_wait的现象，主要原因是某种情况下对方关闭了socket链接，但是我方忙与读或者写，没有关闭连接
1、可以比喻成，客户端通过nginx访问服务，nginx超时时间是10s，如果超过业务服务端执行时间超过10s那么nginx自动关闭，最后业务服务器返回数据给客户端时候（即nginx时候），导致无法连接到服务端所以就报错了

2、客户关掉了浏览器

解决办法：
(1)、除了客户主动关闭游览器动作，我们不用关心，其他问题，比如：修改nginx超时时间，tomcat超时时间，redis超时时间、mysql超时时间等设置长一点，还有将服务端业务避免大业务，尽可能将时间缩短。

(2)、优化系统内核，首先备份/etc/sysctl.conf文件
vim /etc/sysctl.conf

在这个文件中，加入下面的几行内容：
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30

简单的说明下，上面的参数的含义：
net.ipv4.tcp_syncookies = 1 表示开启SYN Cookies。当出现SYN等待队列溢出时，启用cookies来处理，可防范少量SYN攻击，默认为0，表示关闭；
net.ipv4.tcp_tw_reuse = 1 表示开启重用。允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭；
net.ipv4.tcp_tw_recycle = 1 表示开启TCP连接中TIME-WAIT sockets的快速回收，默认为0，表示关闭；
net.ipv4.tcp_fin_timeout 修改系統默认的 TIMEOUT 时间。

在经过这样的调整之后，除了会进一步提升服务器的负载能力之外，还能够防御一定程度的DDoS、CC和SYN攻击，是个一举两得的做法。

此外，如果你的连接数本身就很多，我们可以再优化一下TCP/IP的可使用端口范围，进一步提升服务器的并发能力。依然是往上面的参数文件中，加入下面这些配置：

在这个文件中，加入下面的几行内容：
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 5000

这几个参数，建议只在流量非常大的服务器上开启，会有显著的效果。一般的流量小的服务器上，没有必要去设置这几个参数。这几个参数的含义如下：

net.ipv4.tcp_keepalive_time = 1200 表示当keepalive起用的时候，TCP发送keepalive消息的频度。缺省是2小时，改为20分钟。
net.ipv4.ip_local_port_range = 10000 65000 表示用于向外连接的端口范围。缺省情况下很小：32768到61000，改为10000到65000。（注意：这里不要将最低值设的太低，否则可能会占用掉正常的端口！）
net.ipv4.tcp_max_syn_backlog = 8192 表示SYN队列的长度，默认为1024，加大队列长度为8192，可以容纳更多等待连接的网络连接数。
net.ipv4.tcp_max_tw_buckets = 5000 表示系统同时保持TIME_WAIT的最大数量，如果超过这个数字，TIME_WAIT将立刻被清除并打印警告信息。默认为180000，改为5000。对于Apache、Nginx等服务器，上几行的参数可以很好地减少TIME_WAIT套接字数量，但是对于 Squid，效果却不大。此项参数可以控制TIME_WAIT的最大数量，避免Squid服务器被大量的TIME_WAIT拖死。

参考地址如下：TCP连接出现大量TIME_WAIT的解决办法_咖啡的博客-CSDN博客_端口timewait怎么释放