Trafodion Troubleshooting-org.apache.hadoop.hbase.snapshot.HBaseSnapshotException

现象

在做大表Load数据的时候遇到8448错误，具有错误信息如下，

ERROR[8448] Unable to access Hbase interface. Call to ExpHbaseInterface::doBulkLoad returned error HBASE_DOBULK_LOAD_ERROR(-715). Cause: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot { ss=TRAF_RSRVD_3_TRAFODION.SEABASE.CQGPRS20180730_SNAPSHOT table=TRAF_RSRVD_3:TRAFODION.SEABASE.CQGPRS20180730 type=FLUSH } had an error.  Procedure TRAF_RSRVD_3_TRAFODION.SEABASE.CQGPRS20180730_SNAPSHOT { waiting=[] done=[swrzdn162.cqyd.com,60020,1532924595692, swrzdn123.cqyd.com,60020,1532927440461, swrzdn02.cqyd.com,60020,1532921839386, swrzdn163.cqyd.com,60020,1532929048293, swrzdn124.cqyd.com,60020,1530530102339, swrzdn122.cqyd.com,60020,1532923397158, swrzdn161.cqyd.com,60020,1530530101720, swrzdn121.cqyd.com,60020,1530530102247, swrzdn125.cqyd.com,60020,1530530101579, swrzdn159.cqyd.com,60020,1530530102111, swrzdn03.cqyd.com,60020,1532925830807, swrzdn164.cqyd.com,60020,1530530101723, swrzdn128.cqyd.com,60020,1530530101803, swrzdn110.cqyd.com,60020,1530530102367] }
        at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:368)
        at org.apache.hadoop.hbase.master.MasterRpcServices.isSnapshotDone(MasterRpcServices.java:1022)
        at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55714)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)
Caused by: org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable via swrzdn124.cqyd.com,60020,1530530102339:org.apache.hadoop.hbase.errorhandling.ForeignException$ProxyThrowable: org.apache.hadoop.hbase.NotServingRegionException

分析

通过以上错误信息，发现报错与Snapshot有关，由于目标表较大，在对表做Snapshot的时间超出了配置的timeout设置，导致失败。

解决

调整HBase配置中与Snapshot相关的两个timeout配置，默认为1分钟，可以调为10分钟。

hbase.snapshot.region.timeout
hbase.snapshot.master.timeoutMillis