JDK bug:ciObjectFactory::create_new_metadata：原因完全解析

1、问题

服务不通，接口404，查看日志有一下截图，还有一个更详细的日志

2.详细日志

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f49157f4067, pid=17437, tid=0x00007f48b8ffe700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 1.8.0_92-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x438067]  ciObjectFactory::create_new_metadata(Metadata*)+0x327
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f4910806800):  JavaThread "C1 CompilerThread3" daemon [_thread_in_vm, id=17480, stack(0x00007f48b8efe000,0x00007f48b8fff000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000013

Registers:
RAX=0x000000000000000b, RBX=0x000000000000008a, RCX=0x00000037800182a0, RDX=0x00007f48b8ffea20
RSP=0x00007f48b8ffc300, RBP=0x00007f48b8ffc370, RSI=0x00007f48e0c25ce8, RDI=0x00007f48e0c25ce8
R8 =0x0000000000000089, R9 =0x00007f491638e200, R10=0x0000000000000000, R11=0x000000377f75a290
R12=0x00007f48e0c25ce8, R13=0x00007f48a9ad7240, R14=0x00007f4910806800, R15=0x00007f48a9ad7240
RIP=0x00007f49157f4067, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f48b8ffc300)
0x00007f48b8ffc300:   00007f48a8853080 00007f48b8ffc420
0x00007f48b8ffc310:   00007f48a88513d0 00007f48b8ffd9f0
0x00007f48b8ffc320:   00007f4910008660 00007f49106c4150
0x00007f48b8ffc330:   0000000000000001 00007f48b8ffc400
0x00007f48b8ffc340:   00007f4910806800 000000000000008a
0x00007f48b8ffc350:   00007f48e0c25ce8 00007f48a9ad7240
0x00007f48b8ffc360:   000000000000008a 00007f48a88c38b0
0x00007f48b8ffc370:   00007f48b8ffc3b0 00007f49157f4495
0x00007f48b8ffc380:   00007f48b8ffc3c0 0000000000000000
0x00007f48b8ffc390:   00007f48e0c25ce8 0000000000000000
0x00007f48b8ffc3a0:   00007f491633aaa0 00007f48a88c38b0
0x00007f48b8ffc3b0:   00007f48b8ffc3f0 00007f49157ec167
0x00007f48b8ffc3c0:   0000000000000010 00007f48a88c38b0
0x00007f48b8ffc3d0:   00007f48a8853260 00007f48b94c5c60
0x00007f48b8ffc3e0:   00007f48b8ffc400 00007f48b94c5b30
0x00007f48b8ffc3f0:   00007f48b8ffc530 00007f49157ec478
0x00007f48b8ffc400:   00007f48e0a801c0 00007f48a88532b0
0x00007f48b8ffc410:   00000001000ef3e0 00007f48a88532b0
0x00007f48b8ffc420:   00007f4910806800 00007f48a9ad7240
0x00007f48b8ffc430:   00007f48b8ffc4b0 00007f49157f3dd5
0x00007f48b8ffc440:   00007f48a8852f20 00007f491633aaa0
0x00007f48b8ffc450:   00007f48b8ffc510 00007f48e0bca418
0x00007f48b8ffc460:   00007f48b8ffc4e0 00007f49157e22ce
0x00007f48b8ffc470:   00007f48b8ffc500 00007f48b8ffd9f0
0x00007f48b8ffc480:   00007f48a8852fe0 000000000000006f
0x00007f48b8ffc490:   00000001000ef3e0 00007f48a9ad7240
0x00007f48b8ffc4a0:   0000000000000089 00007f48a9f10ca0
0x00007f48b8ffc4b0:   00007f48b8ffc4f0 00007f49157f44ab
0x00007f48b8ffc4c0:   00007f48b8ffc4e0 0000000000000003
0x00007f48b8ffc4d0:   00000001000ef3e0 0000000000000003
0x00007f48b8ffc4e0:   0000000000000003 00007f48a88532b0
0x00007f48b8ffc4f0:   00007f48b8ffc530 00007f49157ef062 

Instructions: (pc=0x00007f49157f4067)
0x00007f49157f4047:   83 c6 08 39 c7 7c e2 44 8d 48 ff 45 89 08 e9 7b
0x00007f49157f4057:   fd ff ff 66 0f 1f 44 00 00 49 8b 04 24 4c 89 e7
0x00007f49157f4067:   ff 50 08 84 c0 0f 84 05 fd ff ff 41 8b 44 24 08
0x00007f49157f4077:   85 c0 0f 8e f8 fc ff ff 41 f6 84 24 0e 01 00 00 

Register to memory mapping:

RAX=0x000000000000000b is an unknown value
RBX=0x000000000000008a is an unknown value
RCX=0x00000037800182a0: <offset 0x2182a0> in /lib64/libpthread.so.0 at 0x000000377fe00000
RDX=0x00007f48b8ffea20 is pointing into the stack for thread: 0x00007f4910806800
RSP=0x00007f48b8ffc300 is pointing into the stack for thread: 0x00007f4910806800
RBP=0x00007f48b8ffc370 is pointing into the stack for thread: 0x00007f4910806800
RSI=0x00007f48e0c25ce8 is pointing into metadata
RDI=0x00007f48e0c25ce8 is pointing into metadata
R8 =0x0000000000000089 is an unknown value
R9 =0x00007f491638e200: <offset 0xfd2200> in /opt/soft/java/jre/lib/amd64/server/libjvm.so at 0x00007f49153bc000
R10=0x0000000000000000 is an unknown value
R11=0x000000377f75a290: <offset 0x15a290> in /lib64/libc.so.6 at 0x000000377f600000
R12=0x00007f48e0c25ce8 is pointing into metadata
R13=0x00007f48a9ad7240 is an unknown value
R14=0x00007f4910806800 is a thread
R15=0x00007f48a9ad7240 is an unknown value


Stack: [0x00007f48b8efe000,0x00007f48b8fff000],  sp=0x00007f48b8ffc300,  free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x438067]  ciObjectFactory::create_new_metadata(Metadata*)+0x327
V  [libjvm.so+0x438495]  ciObjectFactory::get_metadata(Metadata*)+0x85
V  [libjvm.so+0x430167]  ciSpeculativeTrapData::translate_from(ProfileData const*)+0x67
V  [libjvm.so+0x430478]  ciMethodData::load_extra_data()+0xa8
V  [libjvm.so+0x4325e8]  ciMethodData::load_data()+0x268
V  [libjvm.so+0x424b17]  ciMethod::ensure_method_data(methodHandle)+0xa7
V  [libjvm.so+0x425d67]  ciMethod::ensure_method_data()+0xa7
V  [libjvm.so+0x370e65]  GraphBuilder::try_inline_full(ciMethod*, bool, Bytecodes::Code, Instruction*)+0x545
V  [libjvm.so+0x36d2cf]  GraphBuilder::try_inline(ciMethod*, bool, Bytecodes::Code, Instruction*)+0x11f
V  [libjvm.so+0x36e412]  GraphBuilder::invoke(Bytecodes::Code)+0xbc2
V  [libjvm.so+0x36ed8d]  GraphBuilder::iterate_bytecodes_for_block(int)+0x6ad
V  [libjvm.so+0x37091b]  GraphBuilder::iterate_all_blocks(bool)+0x14b
V  [libjvm.so+0x3712b6]  GraphBuilder::try_inline_full(ciMethod*, bool, Bytecodes::Code, Instruction*)+0x996
V  [libjvm.so+0x36d2cf]  GraphBuilder::try_inline(ciMethod*, bool, Bytecodes::Code, Instruction*)+0x11f
V  [libjvm.so+0x36e412]  GraphBuilder::invoke(Bytecodes::Code)+0xbc2
V  [libjvm.so+0x36ed8d]  GraphBuilder::iterate_bytecodes_for_block(int)+0x6ad
V  [libjvm.so+0x37091b]  GraphBuilder::iterate_all_blocks(bool)+0x14b
V  [libjvm.so+0x3712b6]  GraphBuilder::try_inline_full(ciMethod*, bool, Bytecodes::Code, Instruction*)+0x996
V  [libjvm.so+0x36d2cf]  GraphBuilder::try_inline(ciMethod*, bool, Bytecodes::Code, Instruction*)+0x11f
V  [libjvm.so+0x36e412]  GraphBuilder::invoke(Bytecodes::Code)+0xbc2
V  [libjvm.so+0x36ed8d]  GraphBuilder::iterate_bytecodes_for_block(int)+0x6ad
V  [libjvm.so+0x37091b]  GraphBuilder::iterate_all_blocks(bool)+0x14b
V  [libjvm.so+0x3712b6]  GraphBuilder::try_inline_full(ciMethod*, bool, Bytecodes::Code, Instruction*)+0x996
V  [libjvm.so+0x36d2cf]  GraphBuilder::try_inline(ciMethod*, bool, Bytecodes::Code, Instruction*)+0x11f
V  [libjvm.so+0x36e412]  GraphBuilder::invoke(Bytecodes::Code)+0xbc2
V  [libjvm.so+0x36ed8d]  GraphBuilder::iterate_bytecodes_for_block(int)+0x6ad
V  [libjvm.so+0x3708c2]  GraphBuilder::iterate_all_blocks(bool)+0xf2
V  [libjvm.so+0x3724e7]  GraphBuilder::GraphBuilder(Compilation*, IRScope*)+0x527
V  [libjvm.so+0x37a817]  IRScope::IRScope(Compilation*, IRScope*, int, ciMethod*, int, bool)+0x1e7
V  [libjvm.so+0x37a92f]  IR::IR(Compilation*, ciMethod*, int)+0x9f
V  [libjvm.so+0x358edb]  Compilation::build_hir()+0xdb
V  [libjvm.so+0x35929e]  Compilation::compile_java_method()+0x6e
V  [libjvm.so+0x3593ce]  Compilation::compile_method()+0x4e
V  [libjvm.so+0x35973e]  Compilation::Compilation(AbstractCompiler*, ciEnv*, ciMethod*, int, BufferBlob*)+0x25e
V  [libjvm.so+0x35a4e9]  Compiler::compile_method(ciEnv*, ciMethod*, int)+0xa9
V  [libjvm.so+0x4aea9a]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0xc9a
V  [libjvm.so+0x4afa46]  CompileBroker::compiler_thread_loop()+0x5d6
V  [libjvm.so+0xa6cc5f]  JavaThread::thread_main_inner()+0xdf
V  [libjvm.so+0xa6cd8c]  JavaThread::run()+0x11c
V  [libjvm.so+0x91fad8]  java_start(Thread*)+0x108
C  [libpthread.so.0+0x79d1]


**Current CompileTask:
C1: 319967 19019   !   3       com.mysql.jdbc.Buffer::readString (121 bytes)**


---------------  P R O C E S S  --------------

2.关键日志

1、SIGSEGV (0xb) at pc=0x00007f49157f4067, pid=17437, tid=0x00007f48b8ffe700：中断异常信号量
2、Problematic frame: V [libjvm.so+0x438067] ciObjectFactory::create_new_metadata(Metadata*)+0x327：表示发生的c方法
3、Current thread (0x00007f4910806800): JavaThread “C1 CompilerThread3” daemon [_thread_in_vm, id=17480, stack(0x00007f48b8efe000,0x00007f48b8fff000)]：详细日志中发现当前任务线程：C1 CompilerThread3
4、Current CompileTask:
C1: 319967 19019 ! 3 com.mysql.jdbc.Buffer::readString (121 bytes)：详细日志中发现当前任务线程当前任务

3.结论

【C1编译器】通过线程【C1 CompilerThread3】JIT优化热点方法【com.mysql.jdbc.Buffer::readString】代码时，在调用【libjvm.so】中的代码【ciObjectFactory::create_new_metadata】方法时，【试图访问未分配给自己的内存, 或试图往没有写权限的内存地址写数据】，导致的崩溃

4.JDK：bug

直接去搜索关键字：create_new_metadata，出现一下内容

最终bug链接：

https://bugs.java.com/bugdatabase/view_bug?bug_id=8156137
解释：
I tested with a fix for _previous_version_count, but put in an artificial delay for classes to move from the previous versions list to the deallocate list (simulating on_stack metadata), and I was able to get the same crash, so this is evidence that JDK-8164692 is a separate (but contributing) bug. I’m testing my proposed fix, which is to process previous versions in Klass::clean_weak_klass_links().
翻译：
InstanceKlass::_previous_version_count的问题可能是根本原因。 如果计数错误，则ClassLoaderDataGraph::do_unloading（）可能会跳过调用InstanceKlass::purge_previous_versions（），从而允许ClassLoaderDataGraphKlassIteratorAtomic和G1ConcurrentMark看到这些临时类。
InstanceKlass::_previous_version_count计数错误的一个bug，导致G1ConcurrentMark能调用临时类。

京东遇到过类似bug

链接地址：https://zhuanlan.zhihu.com/p/665260001
很巧合的都是在C1编译jdbc驱动的场景出现的
Current CompileTask:
C1: 319967 19019 ! 3 com.mysql.jdbc.Buffer::readString (121 bytes)
京东：Current CompileTask:
C2:43781461 13331 ! 4 com.mysql.cj.NativeSession::execSQL (579 bytes)

各位大佬如果有更详细的解答可以留言。