使用java查询HBase的优化

最新推荐文章于 2024-09-30 11:46:41 发布

nikkty

最新推荐文章于 2024-09-30 11:46:41 发布

阅读量2.6k

点赞数 1

分类专栏： hbase 文章标签： hbase

本文链接：https://blog.csdn.net/nidayu/article/details/44679415

版权

hbase 专栏收录该内容

0 篇文章 0 订阅

订阅专栏

目前公司的HBase查询速度特别慢，但是从数据库查速度非常快，所以问题应该是出现在java代码的实现上

原来代码：

public Map<String,String> queryByRow(String tableName,String subRowkey){
        HTableInterface table = hBasePool.hTablePool.getTable(tableName); 
        Map<String, String> resultMap = null;
        try {  
        	Scan scan = new Scan();
        	Filter myFilter = new RowFilter(CompareFilter.CompareOp.EQUAL,new SubstringComparator(subRowkey));
        	scan.setFilter(myFilter);
        	long startTime = System.currentTimeMillis();
            ResultScanner scanResult = table.getScanner(scan);
            resultMap = new HashMap<String, String>();  
            String row;
            for (Result res : scanResult) {
          	    row = new String(res.getRow()); 
                for (KeyValue keyValue : res.raw()) { 
                     resultMap.put(row, new String(keyValue.getValue()));    
                }
            }
            return resultMap;
        } catch (IOException e) {  
        	logger.error("hbase过滤查询失败!", e);  
        }
		return null;  
    }

根据准确的rowKey查询，走的却是模糊查询的方法，单个查询最快的永远是使用Get！

现在代码：

public Map<String,String> queryByRow(String tableName,String subRowkey){
    	Get get = new Get(Bytes.toBytes(subRowkey));
        Result result = null;
        Map<String, String> resultMap = new HashMap<String, String>();  
		try {
			result = new HTable(HBasePool.configuration, Bytes.toBytes(tableName)).get(get);
			for (KeyValue kv : result.list()) {
	        	 resultMap.put(subRowkey, new String(Bytes.toString(kv.getValue())));    
	        }
		} catch (Exception e) {
			logger.error("hbase查询失败!", e);  
		}
        return resultMap;
    }

在main方法里调用：

long time = System.currentTimeMillis();
	    System.out.println(MyHBaseDAO.queryByRow("EmailBank", "1066606726@qq.comSPDB1426005427267").toString());
	    System.out.println(System.currentTimeMillis()-time);

使用原来的方法查询需要半分钟，现在查询只要0.5,0.6秒，提升了近500倍的速度！

java链接HBase的方法

HBasePool类：

import org.apache.commons.lang3.math.NumberUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HTablePool;
import org.springframework.stereotype.Service;

import com.lkb.util.InfoUtil;

/**
 * @author baitongyang
 * @date 2014-12-24
 * @Description 返回HtablePool实例的bean类。在此配置hbase属性。
 * @version V1.0
 */
public class HBasePool {
    public static Configuration configuration;
    public static HTablePool hTablePool;
    static {  
        configuration = HBaseConfiguration.create();  
//        configuration.set("hbase.zookeeper.property.clientPort", InfoUtil.getInstance().getInfo("hbase/hbase-setting", "hbase.zookeeper.property.clientPort"));  
        configuration.set("hbase.zookeeper.quorum", InfoUtil.getInstance().getInfo("hbase/hbase-setting", "hbase.zookeeper.quorum"));  
//        configuration.set("hbase.master", InfoUtil.getInstance().getInfo("hbase/hbase-setting", "hbase.master"));
//        configuration.set("hbase.rootdir", InfoUtil.getInstance().getInfo("hbase/hbase-setting", "hbase.rootdir"));
        String ms = InfoUtil.getInstance().getInfo("hbase/hbase-setting", "hbase.maxSize");
        int maxSize = NumberUtils.toInt(ms, 500);
        hTablePool = new HTablePool(configuration, maxSize);
    }
}

hbase-setting.properties:

#hbase zookeeper\u5BA2\u6237\u7AEF\u7AEF\u53E3\u53F7
hbase.zookeeper.property.clientPort=2181
#hbase\u8282\u70B9\u5730\u5740
hbase.zookeeper.quorum=HD1,HD3,HD5
#hbase\u4E3B\u8282\u70B9\u5730\u5740
hbase.master=HD1:60000

#hbase maxSize\u6700\u5927\u8FDE\u63A5\u6570
hbase.maxSize=1000

具体参考LKBHbase项目