hbase client交互过程，插入大量数据，split

最新推荐文章于 2024-08-10 08:27:00 发布

ZJ18712852367

最新推荐文章于 2024-08-10 08:27:00 发布

阅读量923

点赞数

文章标签： hbase

本文链接：https://blog.csdn.net/ZJ18712852367/article/details/81938846

版权

client端交互过程
--------------------
   0.hbase集群启动时，master负责分配区域到指定区域服务器。

   1.联系zk，找出meta表所在rs(regionserver)
       /hbase/meta-region-server

   2.定位row key,找到对应region server

   3.缓存信息在本地。

4.联系RegionServer

5.HRegionServer负责open HRegion对象，为每个列族创建Store对象，Store包含多个StoreFile实例，
他们是对HFile的轻量级封装。每个Store还对应了一个MemStore，用于内存存储数据。

向hbase中插入万条数据：

    @Test
    public void putBigData() throws Exception {
        //格式化
        DecimalFormat format = new DecimalFormat();
        format.applyPattern("00000");
        long start = System.currentTimeMillis() ;
        //创建conf对象
        Configuration conf = HBaseConfiguration.create();
        //通过连接工厂创建连接对象
        Connection conn = ConnectionFactory.createConnection(conf);
        //通过连接查询tableName对象
        TableName tname = TableName.valueOf("ns1:t1");
        HTable table = (HTable)conn.getTable(tname);
        //关闭自动清理缓冲区
        table.setAutoFlush(false);
        for (int i = 4;i<10000;i++){
            Put put = new Put(Bytes.toBytes("row" + format.format(i)));
            //关闭写前日志
            put.setWriteToWAL(false);
            put.addColumn(Bytes.toBytes("f1"),Bytes.toBytes("id"),Bytes.toBytes(i));
            put.addColumn(Bytes.toBytes("f1"),Bytes.toBytes("name"),Bytes.toBytes("tom" + i));
            put.addColumn(Bytes.toBytes("f1"),Bytes.toBytes("age"),Bytes.toBytes(i % 100));
            table.put(put);
            //自定义清理缓冲区
            if(i % 2000 == 0){
                table.flushCommits();
            }
        }
        table.flushCommits();
        System.out.println(System.currentTimeMillis()-start);
    }

关闭写前日志，自定义清理缓冲区。优化效率

log4j:WARN No appenders could be found for logger (org.apache.hadoop.security.Groups).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
4296

统计表


hbase(main):003:0> count 'ns1:t1'
Current count: 1000, row: row01002
Current count: 2000, row: row02002
Current count: 3000, row: row03002
Current count: 4000, row: row04002
Current count: 5000, row: row05002
Current count: 6000, row: row06002
Current count: 7000, row: row07002
Current count: 8000, row: row08002
Current count: 9000, row: row09002
9999 row(s) in 5.6000 seconds

=> 9999

查看hbase的meta（尚未切割）： ns1:t1,,1534732797440.bef8aabb3a68cce8ecb column=info:regioninfo, timestamp=1534902282659, value={ENCODED => bef8aabb3a68cce8ecb7d4c1a64d8578, NAME => 'ns1:t1,,1534
7d4c1a64d8578. 732797440.bef8aabb3a68cce8ecb7d4c1a64d8578.', STARTKEY => '', ENDKEY => ''}

切割表之后再次查看hbase meta

ns1:t1,,1534732797440.bef8aabb3a68cce8ecb column=info:splitA, timestamp=1534904367638, value={ENCODED => beecfcce5ddc1fd889a06653f0fb4555, NAME => 'ns1:t1,,15349043
7d4c1a64d8578. 65500.beecfcce5ddc1fd889a06653f0fb4555.', STARTKEY => '', ENDKEY => 'row05055'}
ns1:t1,,1534732797440.bef8aabb3a68cce8ecb column=info:splitB, timestamp=1534904367638, value={ENCODED => 456e06d350bfa33f97fdebc708470975, NAME => 'ns1:t1,row05055,
7d4c1a64d8578. 1534904365500.456e06d350bfa33f97fdebc708470975.', STARTKEY => 'row05055', ENDKEY => ''}