Trafodion Troubleshooting-java.lang.Out OfMemoryError: unable to create new native thread

现象

EsgynDB节点的sql日志中报错java.lang.Out OfMemoryError: unable to create new native thread，具体信息如下，

2020-03-27 23:22:28,910, ERROR, SQL.EXE, Node Number: 0, CPU: 0, PIN: 45539, Process Name: $Z0000001DG3,,,Java exception in File:../executor/HBaseClient_JNI.cpp, Line:
7329 Method:HTableClient_JNI::fetchRows() TableName:TRAFODION.SCHEMANAME.TABLENAME java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.Out
OfMemoryError: unable to create new native thread
java.util.concurrent.FutureTask.report(FutureTask.java:122)
java.util.concurrent.FutureTask.get(FutureTask.java:192)
org.trafodion.sql.HTableClient.fetchRows(HTableClient.java:1323) Caused by
java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:218)
org.apache.hadoop.hbase.client.ClientSmallScanner.loadCache(ClientSmallScanner.java:259)
org.apache.hadoop.hbase.client.ClientSmallScanner.next(ClientSmallScanner.java:233)
org.apache.hadoop.hbase.client.AbstractClientScanner.next(AbstractClientScanner.java:70)
org.trafodion.sql.HTableClient$ScanHelper.call(HTableClient.java:305)
org.trafodion.sql.HTableClient$ScanHelper.call(HTableClient.java:303)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748) Caused by
java.lang.OutOfMemoryError: unable to create new native thread
java.lang.Thread.start0(Native Method)
java.lang.Thread.start(Thread.java:717)
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:171)
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:322)
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:187)
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:62)
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
org.apache.hadoop.hbase.client.ClientSmallScanner.loadCache(ClientSmallScanner.java:259)
org.apache.hadoop.hbase.client.ClientSmallScanner.next(ClientSmallScanner.java:233)
org.apache.hadoop.hbase.client.AbstractClientScanner.next(AbstractClientScanner.java:70)
org.trafodion.sql.HTableClient$ScanHelper.call(HTableClient.java:305)
org.trafodion.sql.HTableClient$ScanHelper.call(HTableClient.java:303)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

解决

从报错信息看，是因为OutOfMemory导致无法创建native thread，于是怀疑可能是mxosrvr的heap size设置不够，查看ms.env中JVM_MAX_HEAP_SIZE_MB发现值设置为128。

[trafodion@node01 trafodion]$ cat ms.env| grep HEAP
JVM_MAX_HEAP_SIZE_MB=128
ESP_JVM_MAX_HEAP_SIZE_MB=128

改为1024并同步各节点后重启数据库。