问题来源
客户在使用dolphinscheduler调度上传资源时发现jar包上传失败,其他类型文件正常上传
思路三连
前端是否正常上传,服务端是否接受到请求,服务端是否处理请求
前端上传判断
- network看一下请求及报错,请求在,无返回;
- console看一下报错net:ERR_CONNECTION_RESET,连接重置错误;
- postman调用上传接口也没有响应
其他类型能正常上传,只有jar不能上传,暂时认为与前端无关,
服务端是否接受到请求
- 上传接口应用日志没有打印;
- 查看catalina日志,发现catalina日志有报错;请求已经到了服务端,还没有进去就死了。
catalina日志分析
2020-11-03 20:57:59.468 ERROR 414141 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.multipart.MultipartException: Failed to parse multipart servlet request; nested exception is java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadException: java.io.IOException: Connection reset by peer] with root cause
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_181]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_181]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_181]
at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_181]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_181]
at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:163) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.net.NioBlockingSelector.read(NioBlockingSelector.java:177) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.net.NioSelectorPool.read(NioSelectorPool.java:221) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1222) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1141) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:795) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.coyote.http11.Http11InputBuffer.access$300(Http11InputBuffer.java:42) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.coyote.http11.Http11InputBuffer$SocketInputBuffer.doRead(Http11InputBuffer.java:1154) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:102) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.coyote.http11.Http11InputBuffer.doRead(Http11InputBuffer.java:248) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.coyote.Request.doRead(Request.java:555) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:336) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:632) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:362) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:132) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[na:1.8.0_181]
at org.apache.tomcat.util.http.fileupload.util.LimitedInputStream.read(LimitedInputStream.java:132) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.http.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.http.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:881) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at java.io.InputStream.read(InputStream.java:101) ~[na:1.8.0_181]
at org.apache.tomcat.util.http.fileupload.util.Streams.copy(Streams.java:98) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.http.fileupload.util.Streams.copy(Streams.java:68) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.http.fileupload.MultipartStream.readBodyData(MultipartStream.java:572) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.http.fileupload.MultipartStream.discardBodyData(MultipartStream.java:596) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.http.fileupload.MultipartStream.skipPreamble(MultipartStream.java:614) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.http.fileupload.impl.FileItemIteratorImpl.findNextItem(FileItemIteratorImpl.java:213) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.http.fileupload.impl.FileItemIteratorImpl.<init>(FileItemIteratorImpl.java:127) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.http.fileupload.FileUploadBase.getItemIterator(FileUploadBase.java:256) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.http.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:280) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.connector.Request.parseParts(Request.java:2869) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.connector.Request.getParts(Request.java:2771) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.connector.RequestFacade.getParts(RequestFacade.java:1098) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest.parseRequest(StandardMultipartHttpServletRequest.java:95) ~[spring-web-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest.<init>(StandardMultipartHttpServletRequest.java:88) ~[spring-web-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.springframework.web.multipart.support.StandardServletMultipartResolver.resolveMultipart(StandardServletMultipartResolver.java:87) ~[spring-web-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.checkMultipart(DispatcherServlet.java:1178) ~[spring-webmvc-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1012) ~[spring-webmvc-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652) ~[tomcat-embed-core-9.0.39.jar!/:4.0.FR]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) ~[tomcat-embed-core-9.0.39.jar!/:4.0.FR]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.39.jar!/:9.0.39]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.10.RELEASE.jar!/:5.2.10.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) [tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) [tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) [tomcat-embed-core-9.0.39.jar!/:9.0.39]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.39.jar!/:9.0.39]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.39.jar!/:9.0.39]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
en,看不懂 Connection reset by peer 对等方重置连接啥意思?简单理解一下就是请求方重置连接?为什么会这样?
然后去查Connection reset by peer这个问题。
引用:
常见原因:
1)服务器的并发连接数超过了其承载量,服务器会将其中一些连接关闭;
如果知道实际连接服务器的并发客户数没有超过服务器的承载量,则有可能是中了病毒或者木马,引起网络流量异常。可以使用netstat -an查看网络连接情况。
2)客户关掉了浏览器,而服务器还在给客户端发送数据;
3)浏览器端按了Stop;
这两种情况一般不会影响服务器。但是如果对异常信息没有特别处理,有可能在服务器的日志文件中,重复出现该异常,造成服务器日志文件过大,影响服务器的运行。可以对引起异常的部分,使用try…catch捕获该异常,然后不输出或者只输出一句提示信息,避免使用e.printStackTrace();输出全部异常信息。
4)防火墙的问题;
如果网络连接通过防火墙,而防火墙一般都会有超时的机制,在网络连接长时间不传输数据时,会关闭这个TCP的会话,关闭后在读写,就会导致异常。 如果关闭防火墙,解决了问题,需要重新配置防火墙,或者自己编写程序实现TCP的长连接。实现TCP的长连接,需要自己定义心跳协议,每隔一段时间,发送一次心跳协议,双方维持连接。
5)JSP的buffer问题。
JSP页面缺省缓存为8k,当JSP页面数据比较大的时候,有可能JSP没有完全传递给浏览器。这时可以适当调整buffer的大小。
首先排除2)、3)、5),其他文件正常上传排除1)、4)。所以为啥?问了对方网管,客户的网管说这个机器没有做任何特殊处理。
进了tomcat,被截断了,肯定是有地方做了啥操作。
和同事继续研究了一段时间,没啥发现。
客户的网管跑来说,这个请求进入他们网络安全策略。。。好吧 破案了
解决
客户网管将jar包上传的网络安全策略修改,上传正常!
结案语
在遇到问题的时候要一层层细心剖析,找出原因。当我们一时定位不到问题时,也不必纠结,可以找相关的同事帮忙定位。也不必一直卡在问题上,有时候先去处理别的,说不定回过头来突然会有新思路。