机器内存充足，java程序OOM： Native memory allocation (mmap) failed to map xxx bytes for committing reserved

机器内存充足，Java程序却报native内存OOM的问题记录

JVM记录的OOM日志如下：

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
#   JVM is running with Zero Based Compressed Oops mode in which the Java heap is
#     placed in the first 32GB address space. The Java Heap base address is the
#     maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
#     to set the Java Heap base and to place the Java Heap above 32GB virtual address.
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2709), pid=228363, tid=9955
#
# JRE version: OpenJDK Runtime Environment (11.0.1+13) (build 11.0.1+13-LTS)
# Java VM: OpenJDK 64-Bit Server VM (11.0.1+13-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Core dump will be written. Default location: Core dumps may be processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h" (or dumping to /public/home/pengxia/01.Chip_Project/04.capture/02.YX/YX2210/YX_20221026_BZ1/output/core.228363)
#

---------------  S U M M A R Y ------------

Command Line: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.nio.file.spi=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.fs=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.http=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.https=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.ftp=ALL-UNNAMED --add-opens=java.base/sun.net.www.protocol.file=ALL-UNNAMED --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED --illegal-access=deny -Djava.library.path=/opt/gridview/pbs/dispatcher/lib:/usr/local/lib64:/usr/local/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib -Dfile.encoding=UTF-8 -Dcapsule.app=nextflow_21.04.0 -Djava.awt.headless=true -Dcapsule.jar=/public/miniconda3/envs/software/share/nextflow/dist/21.04.0/nextflow-21.04.0-one.jar nextflow.cli.Launcher run /public/home/pengxia/01.Chip_Project/04.capture/02.YX/YX2210/YX_20221026_BZ1/output/DNA_array.nf -params-file /public/home/pengxia/01.Chip_Project/04.capture/02.YX/YX2210/YX_20221026_BZ1/output/DNA_array.run.yml

Host: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz, 20 cores, 125G, CentOS Linux release 7.6.1810 (Core) 
Time: Thu Nov 10 11:30:27 2022 CST elapsed time: 7719 seconds (0d 2h 8m 39s)

---------------  T H R E A D  ---------------

Current thread (0x00002bab6802b800):  JavaThread "Thread-222" [_thread_new, id=9955, stack(0x00002baac3057000,0x00002baac3158000)]

Stack: [0x00002baac3057000,0x00002baac3158000],  sp=0x00002baac3156a80,  free space=1022k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xe34b81]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x161
V  [libjvm.so+0xe3592b]  VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x2b
V  [libjvm.so+0x6e7d58]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xd8
V  [libjvm.so+0xc312ee]  os::pd_commit_memory(char*, unsigned long, bool)+0xce
V  [libjvm.so+0xc2904f]  os::commit_memory(char*, unsigned long, bool)+0x1f
V  [libjvm.so+0xc32a91]  os::pd_create_stack_guard_pages(char*, unsigned long)+0x71
V  [libjvm.so+0xdcef4c]  JavaThread::create_stack_guard_pages()+0x5c
V  [libjvm.so+0xdd0ea4]  JavaThread::run()+0x34
V  [libjvm.so+0xc37082]  thread_native_entry(Thread*)+0xf2

---------------  P R O C E S S  ---------------

Threads class SMR info:
_java_thread_list=0x00002bab68048310, length=21, elements={
0x00002baa34013000, 0x00002baa3444d800, 0x00002baa34452000, 0x00002baa34464800,
0x00002baa34466800, 0x00002baa34469000, 0x00002baa3446b000, 0x00002baa3453b800,
0x00002baa34542800, 0x00002baa35359800, 0x00002baa35409800, 0x00002baa34d14000,
0x00002baa34d15800, 0x00002bab6802c800, 0x00002bab6000c800, 0x00002bab68007000,
0x00002bab6800b800, 0x00002bab68040000, 0x00002bab6c1c8800, 0x00002bab6802b800,
0x00002bab6802d800
}

Java Threads: ( => current thread )
  0x00002baa34013000 JavaThread "main" [_thread_blocked, id=228406, stack(0x00002baa2dd77000,0x00002baa2de78000)]
  0x00002baa3444d800 JavaThread "Reference Handler" daemon [_thread_blocked, id=228413, stack(0x00002baa33d79000,0x00002baa33e7a000)]
  0x00002baa34452000 JavaThread "Finalizer" daemon [_thread_blocked, id=228414, stack(0x00002baa33e7a000,0x00002baa33f7b000)]
  0x00002baa34464800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=228415, stack(0x00002baa4fe75000,0x00002baa4ff76000)]
  0x00002baa34466800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=228416, stack(0x00002baa9b2d0000,0x00002baa9b3d1000)]
  0x00002baa34469000 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=228417, stack(0x00002baa9b3d1000,0x00002baa9b4d2000)]
  0x00002baa3446b000 JavaThread "Sweeper thread" daemon [_thread_blocked, id=228418, stack(0x00002baa9b4d2000,0x00002baa9b5d3000)]
  0x00002baa3453b800 JavaThread "Common-Cleaner" daemon [_thread_blocked, id=228423, stack(0x00002baa9b6d4000,0x00002baa9b7d5000)]
  0x00002baa34542800 JavaThread "Service Thread" daemon [_thread_blocked, id=228424, stack(0x00002baa9b7d5000,0x00002baa9b8d6000)]
  0x00002baa35359800 JavaThread "Thread-0" daemon [_thread_blocked, id=228590, stack(0x00002baaa6e04000,0x00002baaa6f05000)]
  0x00002baa35409800 JavaThread "AnsiLogObserver" [_thread_blocked, id=228777, stack(0x00002baaa6f05000,0x00002baaa7006000)]
  0x00002baa34d14000 JavaThread "Task monitor" [_thread_blocked, id=228808, stack(0x00002bab50905000,0x00002bab50a06000)]
  0x00002baa34d15800 JavaThread "Task submitter" daemon [_thread_blocked, id=228809, stack(0x00002bab50a06000,0x00002bab50b07000)]
  0x00002bab6802c800 JavaThread "process reaper" daemon [_thread_blocked, id=7976, stack(0x00002baa33fbf000,0x00002baa33fe1000)]
  0x00002bab6000c800 JavaThread "Actor Thread 188" daemon [_thread_blocked, id=8404, stack(0x00002baa9b5d3000,0x00002baa9b6d4000)]
  0x00002bab68007000 JavaThread "FileTransfer-thread-144" [_thread_in_native, id=8408, stack(0x00002bab51d15000,0x00002bab51e16000)]
  0x00002bab6800b800 JavaThread "FileTransfer-thread-145" [_thread_blocked, id=8409, stack(0x00002bab50503000,0x00002bab50604000)]
  0x00002bab68040000 JavaThread "Actor Thread 189" daemon [_thread_blocked, id=8413, stack(0x00002bab50b07000,0x00002bab50c08000)]
  0x00002bab6c1c8800 JavaThread "process reaper" daemon [_thread_blocked, id=8417, stack(0x00002baa33f7b000,0x00002baa33f9d000)]
=>0x00002bab6802b800 JavaThread "Thread-222" [_thread_new, id=9955, stack(0x00002baac3057000,0x00002baac3158000)]
  0x00002bab6802d800 JavaThread "Thread-223" [_thread_new, id=9956, stack(0x00002bab50000000,0x00002bab50101000)]

Other Threads:
  0x00002baa34445800 VMThread "VM Thread" [stack: 0x00002baa33c79000,0x00002baa33d79000] [id=228412] _threads_hazard_ptr=0x00002bab68048310
  0x00002baa34545000 WatcherThread [stack: 0x00002baa9b8d8000,0x00002baa9b9d8000] [id=228425]
  0x00002baa34029800 GCTaskThread "GC Thread#0" [stack: 0x00002baa313b6000,0x00002baa314b6000] [id=228407]
  0x00002baab0001000 GCTaskThread "GC Thread#1" [stack: 0x00002baaa7208000,0x00002baaa7308000] [id=228526]
  0x00002baab0002800 GCTaskThread "GC Thread#2" [stack: 0x00002baaa730a000,0x00002baaa740a000] [id=228527]
  0x00002baab0004000 GCTaskThread "GC Thread#3" [stack: 0x00002baaa740c000,0x00002baaa750c000] [id=228528]
  0x00002baab0005800 GCTaskThread "GC Thread#4" [stack: 0x00002baaa750e000,0x00002baaa760e000] [id=228529]
  0x00002baab0007800 GCTaskThread "GC Thread#5" [stack: 0x00002baaa7610000,0x00002baaa7710000] [id=228530]
  0x00002baab0009000 GCTaskThread "GC Thread#6" [stack: 0x00002baaa7712000,0x00002baaa7812000] [id=228531]
  0x00002baab000b000 GCTaskThread "GC Thread#7" [stack: 0x00002baaa7814000,0x00002baaa7914000] [id=228532]
  0x00002baab000c800 GCTaskThread "GC Thread#8" [stack: 0x00002baaa7916000,0x00002baaa7a16000] [id=228533]
  0x00002baab000e800 GCTaskThread "GC Thread#9" [stack: 0x00002baaa7a18000,0x00002baaa7b18000] [id=228534]
  0x00002baab0010800 GCTaskThread "GC Thread#10" [stack: 0x00002baaa7b1a000,0x00002baaa7c1a000] [id=228535]
  0x00002baab0012000 GCTaskThread "GC Thread#11" [stack: 0x00002baaa7c1c000,0x00002baaa7d1c000] [id=228536]
  0x00002baab0014000 GCTaskThread "GC Thread#12" [stack: 0x00002baaa7d1e000,0x00002baaa7e1e000] [id=228537]
  0x00002baab0015800 GCTaskThread "GC Thread#13" [stack: 0x00002baaa7e20000,0x00002baaa7f20000] [id=228538]
  0x00002baab0017000 GCTaskThread "GC Thread#14" [stack: 0x00002baac252c000,0x00002baac262c000] [id=228539]
  0x00002baa34099000 ConcurrentGCThread "G1 Main Marker" [stack: 0x00002baa32886000,0x00002baa32986000] [id=228408]
  0x00002baa3409b000 ConcurrentGCThread "G1 Conc#0" [stack: 0x00002baa32988000,0x00002baa32a88000] [id=228409]
  0x00002baa9c001000 ConcurrentGCThread "G1 Conc#1" [stack: 0x00002baac262e000,0x00002baac272e000] [id=228540]
  0x00002baa9c002800 ConcurrentGCThread "G1 Conc#2" [stack: 0x00002baac2730000,0x00002baac2830000] [id=228541]
  0x00002baa9c004000 ConcurrentGCThread "G1 Conc#3" [stack: 0x00002baac2832000,0x00002baac2932000] [id=228542]

Threads with active compile tasks:

VM state:synchronizing (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00002baa3400f600] Safepoint_lock - owner thread: 0x00002baa34445800
[0x00002baa3400f690] Threads_lock - owner thread: 0x00002baa34445800

Heap address: 0x0000000082000000, size: 30688 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000800000000, Narrow klass shift: 0
Compressed class space size: 1073741824 Address: 0x0000000800000000

Heap:
 garbage-first heap   total 2056192K, used 326341K [0x0000000082000000, 0x0000000800000000)
  region size 4096K, 74 young (303104K), 7 survivors (28672K)
 Metaspace       used 46628K, capacity 50144K, committed 50224K, reserved 1093632K
  class space    used 5332K, capacity 6510K, committed 6528K, reserved 1048576K
Heap Regions: E=young(eden), S=young(survivor), O=old, HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, A=archive, TAMS=top-at-mark-start (previous, next)

...

Card table byte_map: [0x00002baa57bf0000,0x00002baa5b7e0000] _byte_map_base: 0x00002baa577e0000

Marking Bits (Prev, Next): (CMBitMap*) 0x00002baa3407a698, (CMBitMap*) 0x00002baa3407a6d0
 Prev Bits: [0x00002baa5f3d0000, 0x00002baa7d350000)
 Next Bits: [0x00002baa7d350000, 0x00002baa9b2d0000)

Polling page: 0x00002baa2dd43000

Metaspace:

Usage:
  Non-class:     42.61 MB capacity,    40.33 MB ( 95%) used,     2.08 MB (  5%) free+waste,   205.44 KB ( <1%) overhead. 
      Class:      6.36 MB capacity,     5.21 MB ( 82%) used,     1.05 MB ( 17%) free+waste,   101.12 KB (  2%) overhead. 
       Both:     48.97 MB capacity,    45.54 MB ( 93%) used,     3.13 MB (  6%) free+waste,   306.56 KB ( <1%) overhead. 

Virtual space:
  Non-class space:       44.00 MB reserved,      42.67 MB ( 97%) committed 
      Class space:        1.00 GB reserved,       6.38 MB ( <1%) committed 
             Both:        1.04 GB reserved,      49.05 MB (  5%) committed 

Chunk freelists:
   Non-Class:  768 bytes
       Class:  2.25 KB
        Both:  3.00 KB
 
 
CodeHeap 'non-profiled nmethods': size=119172Kb used=7429Kb max_used=8301Kb free=111742Kb
 bounds [0x00002baa48214000, 0x00002baa48a34000, 0x00002baa4f675000]
CodeHeap 'profiled nmethods': size=119172Kb used=19500Kb max_used=22429Kb free=99671Kb
 bounds [0x00002baa40db3000, 0x00002baa423b3000, 0x00002baa48214000]
CodeHeap 'non-nmethods': size=7416Kb used=2195Kb max_used=3970Kb free=5220Kb
 bounds [0x00002baa40675000, 0x00002baa40af5000, 0x00002baa40db3000]
 total_blobs=14000 nmethods=7997 adapters=656
 compilation: enabled
              stopped_count=0, restarted_count=0
 full_count=0
...

 GC Heap History (10 events):
Event: 7.980 GC heap before
{Heap before GC invocations=4 (full 0):
 garbage-first heap   total 2056192K, used 51055K [0x0000000082000000, 0x0000000800000000)
  region size 4096K, 11 young (45056K), 4 survivors (16384K)
 Metaspace       used 34655K, capacity 35696K, committed 35888K, reserved 1081344K
  class space    used 3835K, capacity 4193K, committed 4224K, reserved 1048576K
}
Event: 7.990 GC heap after
{Heap after GC invocations=5 (full 0):
 garbage-first heap   total 2056192K, used 29249K [0x0000000082000000, 0x0000000800000000)
  region size 4096K, 4 young (16384K), 4 survivors (16384K)
 Metaspace       used 34655K, capacity 35696K, committed 35888K, reserved 1081344K
  class space    used 3835K, capacity 4193K, committed 4224K, reserved 1048576K
}
Event: 9.623 GC heap before
{Heap before GC invocations=6 (full 0):
 garbage-first heap   total 2056192K, used 311873K [0x0000000082000000, 0x0000000800000000)
  region size 4096K, 73 young (299008K), 4 survivors (16384K)
 Metaspace       used 36017K, capacity 37152K, committed 37296K, reserved 1083392K
  class space    used 3954K, capacity 4333K, committed 4352K, reserved 1048576K
}
Event: 9.693 GC heap after
{Heap after GC invocations=7 (full 0):
 garbage-first heap   total 2056192K, used 33216K [0x0000000082000000, 0x0000000800000000)
  region size 4096K, 5 young (20480K), 5 survivors (20480K)
 Metaspace       used 36017K, capacity 37152K, committed 37296K, reserved 1083392K
  class space    used 3954K, capacity 4333K, committed 4352K, reserved 1048576K
}
Event: 31.387 GC heap before
{Heap before GC invocations=7 (full 0):
 garbage-first heap   total 2056192K, used 340416K [0x0000000082000000, 0x0000000800000000)
  region size 4096K, 80 young (327680K), 5 survivors (20480K)
 Metaspace       used 44792K, capacity 48051K, committed 48304K, reserved 1091584K
  class space    used 5236K, capacity 6317K, committed 6400K, reserved 1048576K
}
Event: 31.443 GC heap after
{Heap after GC invocations=8 (full 0):
 garbage-first heap   total 2056192K, used 52664K [0x0000000082000000, 0x0000000800000000)
  region size 4096K, 10 young (40960K), 10 survivors (40960K)
 Metaspace       used 44792K, capacity 48051K, committed 48304K, reserved 1091584K
  class space    used 5236K, capacity 6317K, committed 6400K, reserved 1048576K
}
Event: 1539.477 GC heap before
{Heap before GC invocations=8 (full 0):
 garbage-first heap   total 2056192K, used 384440K [0x0000000082000000, 0x0000000800000000)
  region size 4096K, 91 young (372736K), 10 survivors (40960K)
 Metaspace       used 46182K, capacity 49597K, committed 49968K, reserved 1093632K
  class space    used 5314K, capacity 6447K, committed 6528K, reserved 1048576K
}
Event: 1539.588 GC heap after
{Heap after GC invocations=9 (full 0):
 garbage-first heap   total 2056192K, used 56635K [0x0000000082000000, 0x0000000800000000)
  region size 4096K, 7 young (28672K), 7 survivors (28672K)
 Metaspace       used 46182K, capacity 49597K, committed 49968K, reserved 1093632K
  class space    used 5314K, capacity 6447K, committed 6528K, reserved 1048576K
}
Event: 5314.629 GC heap before
{Heap before GC invocations=9 (full 0):
 garbage-first heap   total 2056192K, used 445755K [0x0000000082000000, 0x0000000800000000)
  region size 4096K, 102 young (417792K), 7 survivors (28672K)
 Metaspace       used 46579K, capacity 50075K, committed 50224K, reserved 1093632K
  class space    used 5332K, capacity 6510K, committed 6528K, reserved 1048576K
}
Event: 5314.726 GC heap after
{Heap after GC invocations=10 (full 0):
 garbage-first heap   total 2056192K, used 56005K [0x0000000082000000, 0x0000000800000000)
  region size 4096K, 7 young (28672K), 7 survivors (28672K)
 Metaspace       used 46579K, capacity 50075K, committed 50224K, reserved 1093632K
  class space    used 5332K, capacity 6510K, committed 6528K, reserved 1048576K
}

...

[Global flags]
     intx CICompilerCount                          = 12                                        {product} {ergonomic}
     uint ConcGCThreads                            = 4                                         {product} {ergonomic}
     uint G1ConcRefinementThreads                  = 15                                        {product} {ergonomic}
   size_t G1HeapRegionSize                         = 4194304                                   {product} {ergonomic}
    uintx GCDrainStackTargetSize                   = 64                                        {product} {ergonomic}
   size_t InitialHeapSize                          = 2105540608                                {product} {ergonomic}
   size_t MarkStackSize                            = 4194304                                   {product} {ergonomic}
   size_t MaxHeapSize                              = 32178700288                               {product} {ergonomic}
   size_t MaxNewSize                               = 19306381312                               {product} {ergonomic}
   size_t MinHeapDeltaBytes                        = 4194304                                   {product} {ergonomic}
    uintx NonNMethodCodeHeapSize                   = 7591728                                {pd product} {ergonomic}
    uintx NonProfiledCodeHeapSize                  = 122033256                              {pd product} {ergonomic}
    uintx ProfiledCodeHeapSize                     = 122033256                              {pd product} {ergonomic}
    uintx ReservedCodeCacheSize                    = 251658240                              {pd product} {ergonomic}
     bool SegmentedCodeCache                       = true                                      {product} {ergonomic}
     bool UseCompressedClassPointers               = true                                 {lp64_product} {ergonomic}
     bool UseCompressedOops                        = true                                 {lp64_product} {ergonomic}
     bool UseG1GC                                  = true                                      {product} {ergonomic}

...

---------------  S Y S T E M  ---------------

OS:CentOS Linux release 7.6.1810 (Core) 
uname:Linux 3.10.0-957.el7.x86_64 #1 SMP Sat Dec 15 12:48:25 CST 2018 x86_64
libc:glibc 2.17 NPTL 2.17 
rlimit: STACK infinity, CORE 0k, NPROC 4096, NOFILE 16384, AS infinity, DATA infinity, FSIZE infinity
load average:5.28 4.13 3.43

/proc/meminfo:
MemTotal:       131479676 kB
MemFree:         3327484 kB
MemAvailable:   92729828 kB
Buffers:           13860 kB
Cached:         99082452 kB
SwapCached:       212404 kB
Active:         17107804 kB
Inactive:       100618084 kB
Active(anon):   16262256 kB
Inactive(anon):  6592060 kB
Active(file):     845548 kB
Inactive(file): 94026024 kB
Unevictable:      124808 kB
Mlocked:         3661704 kB
SwapTotal:      33554428 kB
SwapFree:       33266428 kB
Dirty:               604 kB
Writeback:             0 kB
AnonPages:      18679248 kB
Mapped:           308480 kB
Shmem:           4217028 kB
Slab:            1725360 kB
SReclaimable:    1126448 kB
SUnreclaim:       598912 kB
KernelStack:       45616 kB
PageTables:       194156 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    159774916 kB
Committed_AS:   159647480 kB
VmallocTotal:   34359738367 kB
VmallocUsed:     1717088 kB
VmallocChunk:   34289974212 kB
HardwareCorrupted:     0 kB
AnonHugePages:   4749312 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:    76493312 kB
DirectMap2M:    55271424 kB
DirectMap1G:     4194304 kB

/proc/sys/kernel/threads-max (system-wide limit on the number of threads):
1026931

/proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):
100000000

/proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):
229376

container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-19
cpu_memory_nodes: 0-1
active_processor_count: 20
cpu_quota: -1
cpu_period: 100000
cpu_shares: -1
memory_limit_in_bytes: -1
memory_and_swap_limit_in_bytes: -1
memory_soft_limit_in_bytes: -1
memory_usage_in_bytes: 120668778496
memory_max_usage_in_bytes: 0

CPU:total 20 (initial active 20) (10 cores per cpu, 2 threads per core) family 6 model 85 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx, fma
CPU Model and flags from /proc/cpuinfo:
model name    : Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni spec_ctrl intel_stibp flush_l1d arch_capabilities

Memory: 4k page, physical 131479676k(3325992k free), swap 33554428k(33266428k free)

vm_info: OpenJDK 64-Bit Server VM (11.0.1+13-LTS) for linux-amd64 JRE (11.0.1+13-LTS), built on Oct 19 2018 11:38:45 by "tester" with gcc 4.9.2 20150212 (Red Hat 4.9.2-6)

这个Java程序是直接部署在物理机，机器总内存128G，可用内存90多G，可以确定物理内存肯定是充足的
根据信息提示，排除掉内存不足的情况，那这句话就很关键了：

The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
1
还有：

JVM is running with Zero Based Compressed Oops mode in which the Java heap is placed in the first 32GB address space. The Java Heap base address is the maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress to set the Java Heap base and to place the Java Heap above 32GB virtual address.
1
大意是说，在Zero Based Compressed 模式下，Java Heap base address会限制native heap的大小，可以通过修改HeapBaseMinAddress来进行调整

（关于Zero Based Compressed，参考：https://docs.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html）

再看这句：

Heap address: 0x0000000082000000, size: 30688 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
1
结合起来看就是：native heap大小被限制在2G

至于为什么这样，说实话没有找到权威的资料说明。
不过在查找资料的时候找到了这个：https://bugs.openjdk.org/browse/JDK-8266574

大意就是说，开启了CompressedOops的情况下，即使系统有100G内存可用，但是分配给JVM的native内存的也可能只有2G。
目前解决方案是要么禁用CompressedOops，要么通过 -XX:HeapBaseMinAddress来设置Java Heap Base Address，从而调整native heap。

XX:HeapBaseMinAddress的设置比较复杂，简单的方法就是禁用CompressedOops了。
CompressedOops主要是通过压缩指针的方式，达到节省内存的目的，禁用也影响不是很大
通过参数-XX:-UseCompressedOops即可禁用

关于这块的认知还不是很深，后续再进行补充~~

后面找到一个大神的博客：
https://poonamparhar.github.io/out_of_memory_on_64_bit_platform/
这里讲的更清楚了

按照文章里的讲解，观察一些正在运行的jvm进程
通过pmap pid命令进行查看（java -classpath /opt/jdk1.8.0_181/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB pid可以协助查看），发现如下规律

不禁用CompressedOops情况下：

java heap地址位于最前面
xmx设置1G，java heap起始地址3G
xmx设置2G，java heap起始地址2G
xmx设置4G，java heap起始地址28G
xmx设置8G，java heap起始地址24G
xmx设置30G，java heap起始地址2G

禁用CompressedOops情况下：

java heap起始地址不再位于位于最前面，值和xmx的值无关

基本可以看出来，在不禁用CompressedOops情况下：

当xmx小于4G，java heap起始地址为4G-xmx；
当xmx大于4G小于32G，java heap起始地址为32G-xmx
xmx大于32G的情况没测过，不过应该跟禁用CompressedOops的效果是一样的

如果按照这样来说，那设置参数-Xmx4096m，理论上应该也能解决这个OOM问题，待进一步验证