Trafodion Troubleshooting-org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException

现象

使用INSERT或UPSERT插入表报以下错误，但UPSERT USING LOAD或LOAD INTO则执行成功。

*** ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::nextRow returned error HBASE_ACCESS_ERROR(-706). Cause: java.util.concurrent.ExecutionException: java.io.IOException: performScan encountered Exception txID: 72339069014671125 Exception: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: TrxRegionEndpoint coprocessor: getScanner - scanner id 3, Expected nextCallSeq: 4, But the nextCallSeq received from client: 3 in region TRAFODION.ITLR_BAK.TE_SHOPPING_TRADEDATA,,1574649518101.18a64ecc2d634f8f4b08a18eb0f5b28e.,skey=null,ekey=null
 java.util.concurrent.FutureTask.report(FutureTask.java:122)
 java.util.concurrent.FutureTask.get(FutureTask.java:192)
 org.trafodion.sql.HTableClient.fetchRows(HTableClient.java:1325) Caused by 
 java.io.IOException: performScan encountered Exception txID: 72339069014671125 Exception: org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException: TrxRegionEndpoint coprocessor: getScanner - scanner id 3, Expected nextCallSeq: 4, But the nextCallSeq received from client: 3 in region TRAFODION.ITLR_BAK.TE_SHOPPING_TRADEDATA,,1574649518101.18a64ecc2d634f8f4b08a18eb0f5b28e.,skey=null,ekey=null
 org.apache.hadoop.hbase.client.transactional.TransactionalScanner.next(TransactionalScanner.java:327)
 org.apache.hadoop.hbase.client.AbstractClientScanner.next(AbstractClientScanner.java:70)
 org.trafodion.sql.HTableClient$ScanHelper.call(HTableClient.java:307)
 org.trafodion.sql.HTableClient$ScanHelper.call(HTableClient.java:305)
 java.util.concurrent.FutureTask.run(FutureTask.java:266)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748). [2019-11-25 10:56:16]

分析

1. 上述错误“org.apache.hadoop.hbase.exceptions.OutOfOrderScannerNextException”，表示读取超时，可能是HBASE 的 RPC timeout过小。
2. 由于UPSERT USING LOAD或LOAD INTO成功，说明和事务有关。

解决

1. 增加HBase 的RPC TIMEOUT设置。
2. 通过以下CQD限制每次读取的行数。

cqd HBASE_NUM_CACHE_ROWS_MAX '1'
cqd HBASE_NUM_CACHE_ROWS_MIN '1'

3. 通过以下CQD关闭事务。

cqd TRAF_NO_DTM_XN 'ON'