问题描述:hbase库数据迁移为结构化数据,全表遍历时,总是少很多数据;遍历ResultScan正常结束,没有错误,每次都少了近三分之二的数据;
使用的hbase版本:hbase-client-1.0.0, hbase-protocol-1.0.0
全表遍历少数据的原因:使用了ResultScan的iterator的hasNext方法2次(每次遍历)
经过测试:java中的hasNext方法,多次调用不会引起这个问题,就是指针没有移动,调用hasNext时;但hbase中的hasNext方法指针移动了,造成少遍历了;
问题代码:
ResultScanner rs = null;
int count = HBaseUtil.batch;
CountDownLatch latch = new CountDownLatch(HBaseUtil.batch);
rs = HBaseUtil.getResultScanner(HBaseUtil.TEST);
try{
for(Result result : rs){
if(count == HBaseUtil.batch){
logger.info("batch start");
}
TThread thread = new TThread(latch, result);
threadPoolTaskExecutor.execute(thread);
--count;
logger.info("hasNext:" + rs.iterator().hasNext());
if(!rs.iterator().hasNext()){
break;
}
if(count == 0){
try {
logger.info("---------------------await----------------------");
latch.await();
} catch (InterruptedException e) {
logger.error("latch.await exception.");
}finally{
logger.info("====================batch end===============");
count = HBaseUtil.batch;
latch = new CountDownLatch(HBaseUtil.batch);
}
}
}
}catch(Exception e){
logger.error("transferData controller exception:", e);
}finally{
rs.close();
}
去掉hasNext方法的调用,迁移的数据条数就正常了;不知道是hbase的hasNext方法就这样设计的,还是个bug。
找到的源代码,AbstractClientScanner public Iterator<Result> iterator()方法, 它里面的hasNext方法是通过调用next方法来实现的。
@Override
public Iterator<Result> iterator() {
return new Iterator<Result>() {
// The next RowResult, possibly pre-read
Result next = null;
// return true if there is another item pending, false if there isn't.
// this method is where the actual advancing takes place, but you need
// to call next() to consume it. hasNext() will only advance if there
// isn't a pending next().
@Override
public boolean hasNext() {
if (next == null) {
try {
next = AbstractClientScanner.this.next();
return next != null;
} catch (IOException e) {
throw new RuntimeException(e);
}
}
return true;
}
// get the pending next item and advance the iterator. returns null if
// there is no next item.
@Override
public Result next() {
// since hasNext() does the real advancing, we call this to determine
// if there is a next before proceeding.
if (!hasNext()) {
return null;
}
// if we get to here, then hasNext() has given us an item to return.
// we want to return the item and then null out the next pointer, so
// we use a temporary variable.
Result temp = next;
next = null;
return temp;
}
@Override
public void remove() {
throw new UnsupportedOperationException();
}
};
}
当时从页面写的一个控制器迁移hbase历史数据,莫名的多了一倍的问题??
当时部署了2台机器,使用的ngix负载均衡,在页面点击触发后台迁移程序后,一直没有返回结果,等待迁移程序跑完才行(可能要很久),可能ngix认为这个请求超时了,就把迁移程序的请求,转发到了另一台机器,相当于迁移程序跑2遍。数据又翻倍了。一波三折。