查询hbase1中表d是报异常,如下:
Tue Feb 02 13:48:50 CST 2016, RpcRetryingCaller{globalStartTime=1454392129833, pause=100, retries=3}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region d,,1451161931525.ebebcda3b62fbe7c7bce561d24f64df2. is not online on datanode2,16020,1453890442843
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2898)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:947)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2234)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region d,,1451161931525.ebebcda3b62fbe7c7bce561d24f64df2. is not online on datanode2,16020,1453890442843
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2898)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:947)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2234)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:322)
at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:380)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
... 4 more
定位问题:看hbase程序是否挂掉(没有),查询其他表看是否报异常(没有),可以看出只是查该表才会出问题,该表的数据量比较大,所以怀疑表d在hbase中出了问题(该表需要经常分裂,无法预知rowkey的内容,所以只能交给hbase做分裂)。
了解一下hbase查询数据的流程,到.META中查询表的region server信息,然后再去对应的region上去取数据,看上面报的异常中有:Region d,,1451161931525.ebebcda3b62fbe7c7bce561d24f64df2. is not online on datanode2,再看代码中hBaseAdmin.tableExists(tableName)返回true,
初步可以判断表d的meta信息出现了问题,看看hbase的ui界面中Tables那个单元中也没有d表,所以决定把表信息删掉再重建。
// 初始化配置
static {
conf = HBaseConfiguration.create();
try {
connection = ConnectionFactory.createConnection(conf);
} catch (IOException e) {
logger.error("", e);
}
}
public static void dropTable(String tableName) {
HBaseAdmin hBaseAdmin = null;
try {
hBaseAdmin = (HBaseAdmin) connection.getAdmin();
if (hBaseAdmin.tableExists(tableName)) {
hBaseAdmin.disableTable(tableName);
hBaseAdmin.deleteTable(tableName);
System.out.println(tableName + " is deleted....");
return ;
}
HTableDescriptor tableDescriptor = new HTableDescriptor(TableName.valueOf(tableName));
tableDescriptor.addFamily(new HColumnDescriptor(Constants.FAMILY1));
hBaseAdmin.createTable(tableDescriptor);
} catch (Exception e) {
logger.error("", e);
} finally {
if(hBaseAdmin != null)
try {
hBaseAdmin.close();
} catch (IOException e) {
logger.error("", e);
}
}
}
执行上面代码后是成功了,表的数据也没有丢,所以猜测是.meta给出的表d的region信息是错误的,把.meta的信息重构了一下。
上面只是解决问题,具体内部细节需要慢慢深入学习hbase了~~~