记一次线上Zip文件操作导致JVM奔溃

又好久没有整理笔记了。这次疫情防控正好周末居家，就想着花点时间整理下自己工作中遇到一些疑难杂症，帮助自己更透彻的理解某些问题！

  言归正传，问题的主要经过是这样的：生产上面运营人员导入一批表盘文件，系统直接挂了。运维人员找到我，整理错误日志，部分如下：

 hs.err.pid1.log文件部分如下

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f4518bbe84e, pid=1, tid=0x00007f44ead03b10
#
# JRE version: OpenJDK Runtime Environment (8.0_212-b04) (build 1.8.0_212-b04)
# Java VM: OpenJDK 64-Bit Server VM (25.212-b04 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 3.12.0
# Distribution: Custom build (Sat May  4 17:33:35 UTC 2019)
# Problematic frame:
# C  [libzip.so+0x584e]
#
# Core dump written. Default location: //core or core.1
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   https://icedtea.classpath.org/bugzilla
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x000055e3dad05000):  JavaThread "http-nio-8095-exec-6" daemon [_thread_in_native, id=58, stack(0x00007f44eac03000,0x00007f44ead03ad0)]

siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr: 0x00007f44f50ed821

Registers:
RAX=0x000055e3da29aa40, RBX=0x000055e3de015aa0, RCX=0x0000000000000740, RDX=0xffb8007800000003
RSP=0x00007f44eacfd940, RBP=0x00007f44f50ed801, RSI=0x0000000000000023, RDI=0x0000000000000007
R8 =0x0000000000000061, R9 =0x000000000000000e, R10=0x00007f4509f04a18, R11=0x00007f4509f049d8
R12=0x00007f44eacfd9f0, R13=0x00007f44f50ed801, R14=0x00000000afef5055, R15=0x000055e3da29aa40
RIP=0x00007f4518bbe84e, EFLAGS=0x0000000000010202, CSGSFS=0x8be2000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f44eacfd940)
0x00007f44eacfd940:   00000000afef4fe5 00000000de015b80
0x00007f44eacfd950:   000000000000000e 00007f451938fa84
0x00007f44eacfd960:   00007f44eacfd9fe 000055e3de015c20
0x00007f44eacfd970:   000000000000000e 00007f44eacfd9f0
0x00007f44eacfd980:   0000000000000000 00000000afef5055
0x00007f44eacfd990:   000055e3de015aa0 00007f4518bbf47e
0x00007f44eacfd9a0:   00007f44eacfd9c8 00007f450000000e
0x00007f44eacfd9b0:   000055e3dad055c0 000000000000000e
0x00007f44eacfd9c0:   000055e3dad051e0 0000000000000000
0x00007f44eacfd9d0:   00007f44eacfde48 000055e3de015aa0
0x00007f44eacfd9e0:   00007f44eacfd9f0 00007f4518bbd5d7
0x00007f44eacfd9f0:   6863746177353734 000070697a2e3833
0x00007f44eacfda00:   00007f44eacfde20 00007f4509bd13db
0x00007f44eacfda10:   00007f44eacfde78 00007f44eacfdc08
0x00007f44eacfda20:   00007f44eacfdad8 000055e3dad05000
0x00007f44eacfda30:   000055e3dad05000 00007f44eacfdad8
0x00007f44eacfda40:   00007f44eacfda90 00007f44eacfdc08
0x00007f44eacfda50:   000055e3dad05000 00007f45192a3dbc
0x00007f44eacfda60:   00007f44eacfdab0 000055e3dad051e0
0x00007f44eacfda70:   00007f45192a9d64 00007f44eacfdb18
0x00007f44eacfda80:   000055e3da9119b0 000055e3da911e08
0x00007f44eacfda90:   00007f44f6c47e48 000055e3dad05000
0x00007f44eacfdaa0:   00007f44f6c47e48 000055e3dad05000
0x00007f44eacfdab0:   000055e3dad05580 000055e3da9119a0
0x00007f44eacfdac0:   000055e3da9119b0 000055e3da911d88
0x00007f44eacfdad0:   00000000000003d8 00007f44f6c47e48
0x00007f44eacfdae0:   000055e3dad05000 00000000d3e16b18
0x00007f44eacfdaf0:   00007f44eacfdf10 00007f450a1cc38c
0x00007f44eacfdb00:   00007f450b25ebdc 00007f44f6c47e48
0x00007f44eacfdb10:   000055e3dad05000 00000007843d2280
0x00007f44eacfdb20:   00000007843d4f78 00000005d3e16b18
0x00007f44eacfdb30:   00007f44eacfdae0 00007f44eacfdb21 

Instructions: (pc=0x00007f4518bbe84e)
0x00007f4518bbe82e:   00 00 00 48 8b 6d 00 48 c7 40 28 00 00 00 00 0f
0x00007f4518bbe83e:   84 5d 01 00 00 48 2b 6b 28 48 03 6b 18 49 89 ed
0x00007f4518bbe84e:   41 0f b7 45 20 45 0f b7 75 1c 41 0f b7 6d 1e 66
0x00007f4518bbe85e:   89 44 24 16 4d 89 f4 89 44 24 10 41 8b 45 0c 66 

Register to memory mapping:

RAX=0x000055e3da29aa40 is an unknown value
RBX=0x000055e3de015aa0 is an unknown value
RCX=0x0000000000000740 is an unknown value
RDX=0xffb8007800000003 is an unknown value
RSP=0x00007f44eacfd940 is pointing into the stack for thread: 0x000055e3dad05000
RBP=0x00007f44f50ed801 is an unknown value
RSI=0x0000000000000023 is an unknown value
RDI=0x0000000000000007 is an unknown value
R8 =0x0000000000000061 is an unknown value
R9 =0x000000000000000e is an unknown value
R10=0x00007f4509f04a18 is at entry_point+88 in (nmethod*)0x00007f4509f04850
R11=0x00007f4509f049d8 is at entry_point+24 in (nmethod*)0x00007f4509f04850
R12=0x00007f44eacfd9f0 is pointing into the stack for thread: 0x000055e3dad05000
R13=0x00007f44f50ed801 is an unknown value
R14=0x00000000afef5055 is an unknown value
R15=0x000055e3da29aa40 is an unknown value


Stack: [0x00007f44eac03000,0x00007f44ead03ad0],  sp=0x00007f44eacfd940,  free space=1002k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libzip.so+0x584e]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 986  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007f4509f04a18 [0x00007f4509f049c0+0x58]
j  java.util.zip.ZipFile.getInputStream(Ljava/util/zip/ZipEntry;)Ljava/io/InputStream;+88
J 16467 C1 com.idoocloud.admin.console.service.impl.DeviceFaceServiceImpl.readZip(Ljava/lang/String;Ljava/util/Map;Ljava/util/Map;)V (928 bytes) @ 0x00007f450c2fa96c [0x00007f450c2f8320+0x264c]
j  com.idoocloud.admin.console.service.impl.DeviceFaceServiceImpl.readZip(Ljava/lang/String;Ljava/util/Map;Ljava/util/Map;)V+650
j  com.idoocloud.admin.console.service.impl.DeviceFaceServiceImpl.parseZip(Ljava/io/File;Lcom/idoocloud/admin/console/response/DeviceFaceImportSumResult;Lcn/hutool/core/map/TableMap;)V+48
j  com.idoocloud.admin.console.service.impl.DeviceFaceServiceImpl.parseFile(Lcom/idoocloud/admin/console/request/ImportDeviceFaceRequest;Lcom/idoocloud/common/authc/UserToken;)Lcom/idoocloud/admin/console/response/DeviceFaceImportSumResult;+107
j  com.idoocloud.admin.console.web.OtaFaceController.batchImport(Lorg/springframework/web/multipart/commons/CommonsMultipartFile;Lorg/springframework/web/multipart/commons/CommonsMultipartFile;Lorg/springframework/web/multipart/commons/CommonsMultipartFile;Ljava/lang/Long;Ljava/lang/Integer;Lcom/idoocloud/common/authc/UserToken;)Lcom/idoocloud/common/ApiResult;+76
j  com.idoocloud.admin.console.web.OtaFaceController$$FastClassBySpringCGLIB$$f642c964.invoke(ILjava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+152
j  org.springframework.cglib.proxy.MethodProxy.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+19
j  org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint()Ljava/lang/Object;+19
J 13664 C1 org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()Ljava/lang/Object; (126 bytes) @ 0x00007f450bd18044 [0x00007f450bd17660+0x9e4]
j  org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(Lorg/aopalliance/intercept/MethodInvocation;)Ljava/lang/Object;+7
J 13664 C1 org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()Ljava/lang/Object; (126 bytes) @ 0x00007f450bd17f94 [0x00007f450bd17660+0x934]
j  org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Lorg/aopalliance/intercept/MethodInvocation;)Ljava/lang/Object;+18
J 13664 C1 org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()Ljava/lang/Object; (126 bytes) @ 0x00007f450bd17f94 [0x00007f450bd17660+0x934]
J 14406 C1 org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;Lorg/springframework/cglib/proxy/MethodProxy;)Ljava/lang/Object; (231 bytes) @ 0x00007f450bf6980c [0x00007f450bf68fe0+0x82c]
j  com.idoocloud.admin.console.web.OtaFaceController$$EnhancerBySpringCGLIB$$f68f64d9.batchImport(Lorg/springframework/web/multipart/commons/CommonsMultipartFile;Lorg/springframework/web/multipart/commons/CommonsMultipartFile;Lorg/springframework/web/multipart/commons/CommonsMultipartFile;Ljava/lang/Long;Ljava/lang/Integer;Lcom/idoocloud/common/authc/UserToken;)Lcom/idoocloud/common/ApiResult;+60
v  ~StubRoutines::call_stub
J 2609  sun.reflect.NativeMethodAccessorImpl.invoke0(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (0 bytes) @ 0x00007f450a3b5037 [0x00007f450a3b4fc0+0x77]
J 13426 C2 sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (104 bytes) @ 0x00007f450bc695f4 [0x00007f450bc69580+0x74]
J 3246 C2 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (62 bytes) @ 0x00007f450a699268 [0x00007f450a6991c0+0xa8]
j  org.springframework.web.method.support.InvocableHandlerMethod.doInvoke([Ljava/lang/Object;)Ljava/lang/Object;+16
j  org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(Lorg/springframework/web/context/request/NativeWebRequest;Lorg/springframework/web/method/support/ModelAndViewContainer;[Ljava/lang/Object;)Ljava/lang/Object;+56
j  org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(Lorg/springframework/web/context/request/ServletWebRequest;Lorg/springframework/web/method/support/ModelAndViewContainer;[Ljava/lang/Object;)V+4
j  org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/springframework/web/method/HandlerMethod;)Lorg/springframework/web/servlet/ModelAndView;+244
j  org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Lorg/springframework/web/method/HandlerMethod;)Lorg/springframework/web/servlet/ModelAndView;+81
j  org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljava/lang/Object;)Lorg/springframework/web/servlet/ModelAndView;+7
j  org.springframework.web.servlet.DispatcherServlet.doDispatch(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+257
j  org.springframework.web.servlet.DispatcherServlet.doService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+212
j  org.springframework.web.servlet.FrameworkServlet.processRequest(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+71
j  org.springframework.web.servlet.FrameworkServlet.doPost(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3
j  javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+149
j  org.springframework.web.servlet.FrameworkServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+33
j  javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V+30

通过日志，我们不难发现是程序在使用java.util.zip.ZipFile.getEntry方法时出现了问题。带着疑惑，我将关键信息在搜索引擎查询了一番，好像是jvm的mmap机制引起的。既然如此，jdk官方应该有记录吧，带着疑惑，查询官网，找到相关的Issue记录，得到如下解释[JDK-8190772] Crash in [libzip.so+0x3bec] Java_java_util_zip_ZipFile_getEntry+0xfc - Java Bug System

 这问题在jdk9种得到了彻底解决，大致意思是当jvm的参数设置为true时-Dsun.zip.disableMemoryMapping=true， ZIP_GetEntry在文件被移动或者覆盖的时候会产生奔溃的bug。查找一篇中文对此问题的记录如下：

在大多数情况下，当正在访问的 jar 文件在 JVM 实例运行时被修改/覆盖时，ZIP_GetEntry 发生崩溃。出于性能原因，HotSpot JVM 内存使用 mmap 映射每个 Jar 文件的中央目录结构。这样做是为了避免每次需要从 Jar 文件中读取条目时从磁盘读取中央目录结构数据。当修改或覆盖磁盘上的 Jar 文件时，先前读取的数据的 JVM 副本与磁盘上的 jar 文件不一致，并且任何尝试从修改后的 jar 中读取和加载条目都可能导致应用程序崩溃。从 1.6.0_23 开始，可以使用一个属性来禁用 Jar 文件的中央目录结构的内存映射：

-Dsun.zip.disableMemoryMapping = true

至此，该问题的答案我们已经找到了。修改jvm启动参数重启项目，对比修改前重现的崩溃bug，问题终于得到了解决。

总结：我们在做zip文件操作的过程中，最好保证文件不要被其他逻辑更改。同时，必要时还是要对jvm常见参数以及java常见虚拟机有一定的了解，从而避免这种生产事故的发生。

附上mmap被禁止带来的影响：

请注意，启用此属性会对应用程序产生一些性能影响，因为 JVM 需要在每次读取 Jar 文件条目时一次又一次从磁盘上的 Jar 文件中读取中央目录结构。因此，最好确保在 JVM 加载了映像文件时，不会修改或覆盖 jar 文件。在 Java 9 中，此 ZIP 崩溃已通过以下增强功能得以解决： JDK-8142508：将 juzZipFile 的本机实现引入 Java 中，以消除昂贵的 jni 成本和 mmap 崩溃风险

最后还有一个问题，大家可以去思考验证下：我在做奔溃bug复现的时候，本地Windows系统无论是使用idea调试还是直接打jar包在cmd控制台运行都不会奔溃，但是一上Linux系统就奔溃。这里面我觉得最大的可能还是Linux系统的文件读写机制和Windows不同（同时我这里面还有一个变量，我本地使用的jdk和Linux系统打包docker环境使用的open jdk不知道会不会有影响），感兴趣的朋友可以去验证一下！

最后附上查阅的相关资料，希望能对大家有帮助：

https://cloud.tencent.com/developer/article/1039901

https://zgserver.com/jvmjava-util-zip-zipfile-getentry.html

https://www.codenong.com/jsc29d3743d101/