cassandra节点异常：java.lang.OutOfMemoryError: Direct buffer memory

宕机两次了：

频繁发送多次请求到cassandra集群， 整个集群down掉， 查看日志发现：

debug.log文件：

ERROR [ReadStage-146] 2017-09-13 10:40:08,415 JVMStabilityInspector.java:141 - JVM state determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_111]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.8.0_111]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_111]
at org.apache.cassandra.utils.memory.BufferPool.allocate(BufferPool.java:109) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.utils.memory.BufferPool.access$1000(BufferPool.java:46) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.utils.memory.BufferPool$LocalPool.allocate(BufferPool.java:406) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.utils.memory.BufferPool$LocalPool.access$000(BufferPool.java:333) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:121) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:93) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:156) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:40) ~[apache-cassandra-3.9.jar:3.9]
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949) ~[caffeine-2.2.6.jar:na]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807) ~[caffeine-2.2.6.jar:na]
at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1892) ~[na:1.8.0_111]

system.log文件：

WARN  [ReadStage-138] 2017-09-13 10:40:08,414 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[ReadStage-138,5,main]: {}
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_111]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[na:1.8.0_111]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_111]
at org.apache.cassandra.utils.memory.BufferPool.allocate(BufferPool.java:109) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.utils.memory.BufferPool.access$1000(BufferPool.java:46) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.utils.memory.BufferPool$LocalPool.allocate(BufferPool.java:406) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.utils.memory.BufferPool$LocalPool.access$000(BufferPool.java:333) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:121) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:93) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:156) ~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:40) ~[apache-cassandra-3.9.jar:3.9]
at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949) ~[caffeine-2.2.6.jar:na]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807) ~[caffeine-2.2.6.jar:na]
at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1892) ~[na:1.8.0_111]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805) ~[caffeine-2.2.6.jar:na]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788) ~[caffeine-2.2.6.jar:na]
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97) ~[caffeine-2.2.6.jar:na]

"Run services with -Djdk.nio.maxCachedBufferSize=262144 to avoid this problem"

ref: https://issues.apache.org/jira/browse/CASSANDRA-10689

from email: 老外的解释，好像比较合理

There is a "kinda leak" in jvm around this you may run into, can try with -Djdk.nio.maxCachedBufferSize=262144 if above 8u102. You can also try increasing the size allowed for direct byte buffers. It defaults to size of heap -XX:MaxDirectMemorySize=?G

Some NIO channel operations use temporary DirectByteBuffers which are cached in thread-local caches to avoid having to allocate / free a buffer at every operation. 
Unfortunately, there is no bound imposed on the size of buffers added to the thread-local caches. So, infrequent channel operations that require a very large buffer can create a native memory leak. 

Ability to limit the capacity of buffers that can be held in the temporary buffer cache
The system property  jdk.nio.maxCachedBufferSize  has been introduced in 8u102 to limit the memory used by the "temporary buffer cache." The temporary buffer cache is a per-thread cache of direct memory used by the NIO implementation to support applications that do I/O with buffers backed by arrays in the java heap. The value of the property is the maximum capacity of a direct buffer that can be cached. If the property is not set, then no limit is put on the size of buffers that are cached. Applications with certain patterns of I/O usage may benefit from using this property. In particular, an application that does I/O with large multi-megabyte buffers at startup but does I/O with small buffers may see a benefit to using this property. Applications that do I/O using direct buffers will not see any benefit to using this system property. 
See  JDK-8147468

java进程内存使用：heap_space 和 off_heap_space(堆外内存)

Direct buffer memory属于off_heap_space(堆外内存)， 内存不够使用，出现OOM错误

http://blog.csdn.net/lindev/article/details/53261897

http://blog.csdn.net/zshake/article/details/46785469

http://www.zhimengzhe.com/shujuku/other/263521.html