连接hbase_HBase入门

HBase在大数据生态中的地位举足轻重，它是谷歌bigtable的开源实现，是一种分布式存储的NoSQL数据库，能自动分片和故障转移，与HDFS高度集成，适合海量数据的高效查询。我目前用过的业务场景包括：

1.存储日志数据

2.存储车辆GPS数据，设备上报数据

3.kafka tpoic的offset

HBase架构

我们可以通过HBase的web管理界面来直观感受HBase的架构

1.HBase依赖Zookeeper，Zookeeper存储其元数据，对Master和RegionServer进行分布式协调

2.HDFS作为HBase运行的底层文件系统

3.RegionServer为从节点，是数据节点，存储数据

4.Master RegionServer需要实时地向Master上报运行状况

常用HBase Shell命令

查看帮助信息

[root@cdh3 ~]# hbase shellHBase ShellUse "help" to get list of supported commands.Use "exit" to quit this interactive shell.For Reference, please visit: http://hbase.apache.org/2.0/book.html#shellVersion 2.2.3.7.1.3.0-100, rUnknown, Wed Aug  5 10:49:56 UTC 2020Took 0.0012 seconds                                                                                                                  hbase(main):001:0> helpHBase Shell, version 2.2.3.7.1.3.0-100, rUnknown, Wed Aug  5 10:49:56 UTC 2020Type 'help "COMMAND"', (e.g. 'help "get"' -- the quotes are necessary) for help on a specific command.Commands are grouped. Type 'help "COMMAND_GROUP"', (e.g. 'help "general"') for help on a command group.COMMAND GROUPS:  Group name: general  Commands: processlist, status, table_help, version, whoami  Group name: ddl  Commands: alter, alter_async, alter_status, clone_table_schema, create, describe, disable, disable_all, drop, drop_all, enable, enable_all, exists, get_table, is_disabled, is_enabled, list, list_regions, locate_region, show_filters  Group name: namespace  Commands: alter_namespace, create_namespace, describe_namespace, drop_namespace, list_namespace, list_namespace_tables  Group name: dml  Commands: append, count, delete, deleteall, get, get_counter, get_splits, incr, put, scan, truncate, truncate_preserve

创建一个表，必须要指定表名称和列簇名

hbase(main):002:0> create 'test', 'cf'Created table testTook 1.8893 seconds                                                                       => Hbase::Table - test

列出表信息

hbase(main):003:0> list 'test'TABLE                                                                                      test                                                                                      1 row(s)Took 0.0231 seconds                                                                       => ["test"]

查看表详细信息，使用describe命令

hbase(main):004:0> describe 'test'Table test is ENABLED                                                                     test                                                                                      COLUMN FAMILIES DESCRIPTION                                                                {NAME => 'cf', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}                                                             1 row(s)QUOTAS                                                                                    0 row(s)Took 0.2339 seconds

添加数据

hbase(main):005:0> put 'test','row1','cf:a','value1'Took 0.1284 seconds                                                                       hbase(main):006:0> put 'test','row2','cf:b','value2'Took 0.0048 seconds                                                                        hbase(main):007:0> put 'test','row3','cf:c','value3'Took 0.0085 seconds 

查看表中的所有数据

hbase(main):008:0> scan 'test'ROW                                COLUMN+CELL                                          row1                              column=cf:a, timestamp=1600003697537, value=value1     row2                              column=cf:b, timestamp=1600003711262, value=value2      row3                              column=cf:c, timestamp=1600003732296, value=value3     3 row(s)Took 0.0235 seconds    

获取单行的数据

hbase(main):009:0> get 'test','row1'COLUMN                             CELL                                                   cf:a                              timestamp=1600003697537, value=value1                 1 row(s)Took 0.0139 second

HBase Java常用API

并以创建表为例进行测试

import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.hbase.HBaseConfiguration;import org.apache.hadoop.hbase.HColumnDescriptor;import org.apache.hadoop.hbase.HTableDescriptor;import org.apache.hadoop.hbase.TableName;import org.apache.hadoop.hbase.client.*;import org.apache.hadoop.hbase.util.Bytes;import java.io.IOException;import java.lang.reflect.Field;import java.text.SimpleDateFormat;import java.util.*;public class HBaseUtil {    private static Connection connection = null;    /**     * 初始化hbase的连接     *     * @throws IOException     */    private static void initConnection() throws IOException {        if (connection == null || connection.isClosed()) {            Configuration conf = HBaseConfiguration.create();            conf.set("hbase.zookeeper.quorum", "192.168.0.171,192.168.0.207,192.168.0.208");            conf.set("hbase.zookeeper.property.clientPort", "2181");            connection = ConnectionFactory.createConnection(conf);        }    }    /**     * 获得连接     *     * @return     * @throws IOException     */    public static Connection getConnection() throws IOException {        if (connection == null || connection.isClosed()) {            initConnection();        }        //连接可用直接返回连接        return connection;    }    /**     * 创建表     *     * @param tableNameString     * @param columnFamily     * @throws IOException     */    public static void createTable(Connection connection, String tableNameString, String columnFamily) throws IOException {        Admin admin = connection.getAdmin();        TableName tableName = TableName.valueOf(tableNameString); //d2h (data to HBase)        HTableDescriptor table = new HTableDescriptor(tableName);        HColumnDescriptor family = new HColumnDescriptor(columnFamily);        table.addFamily(family);        //判断表是否已经存在        if (admin.tableExists(tableName)) {            admin.disableTable(tableName);            admin.deleteTable(tableName);        }        admin.createTable(table);    }    /**     * 判断hbase的表是否存在     *     * @param tableName     * @return     * @throws Exception     */    public static boolean tableExists(String tableName) throws Exception {        if (connection == null || connection.isClosed()) {            initConnection();        }        Admin admin = connection.getAdmin();        if (admin.tableExists(TableName.valueOf(tableName))) {            return true;        }        return false;    }    public static Table getTable(String tableName)throws Exception{        Connection connection = getConnection();        return connection.getTable(TableName.valueOf(tableName));    }    /**     * 获取插入HBase的操作put     *     * @param rowKeyString     * @param familyName     * @param columnName     * @param columnValue     * @return     */    public static Put createPut(String rowKeyString, byte[] familyName, String columnName, String columnValue) {        byte[] rowKey = rowKeyString.getBytes();        Put put = new Put(rowKey);        put.addColumn(familyName, columnName.getBytes(), columnValue.getBytes());        return put;    }    /**     * 获取插入HBase的操作put     *     * @param rowKeyString     * @param familyName     * @param columns      列     * @return     */    public static Put createPut(String rowKeyString, byte[] familyName, Map columns) {        byte[] rowKey = rowKeyString.getBytes();        Put put = new Put(rowKey);        for (Map.Entry entry : columns.entrySet()) {            put.addColumn(familyName, entry.getKey().getBytes(), entry.getValue().getBytes());        }        return put;    }        /**     * 打印HBase查询结果     *     * @param result     */    public static void print(Result result) {        //result是个四元组        byte[] row = result.getRow(); //行键        NavigableMap<byte[], NavigableMap<byte[], NavigableMapbyte[]>>> map = result.getMap();        for (Map.Entry<byte[], NavigableMap<byte[], NavigableMapbyte[]>>> familyEntry : map.entrySet()) {            byte[] familyBytes = familyEntry.getKey(); //列族            for (Map.Entry<byte[], NavigableMapbyte[]>> entry : familyEntry.getValue().entrySet()) {                byte[] column = entry.getKey(); //列                for (Map.Entrybyte[]> longEntry : entry.getValue().entrySet()) {                    Long time = longEntry.getKey(); //时间戳                    byte[] value = longEntry.getValue(); //值                    System.out.println(String.format("行键rowKey=%s,列族columnFamily=%s,列column=%s,时间戳timestamp=%d,值value=%s", new String(row), new String(familyBytes), new String(column), time, new String(value)));                }            }        }    }    // 测试创建hbase_test表    public static void main(String[] args) throws Exception {        Connection connection = HBaseUtil.getConnection();        Boolean tableExists = HBaseUtil.tableExists("hbase_test");        System.out.println(tableExists);        HBaseUtil.createTable(connection, "hbase_test", "cf1");        tableExists = HBaseUtil.tableExists("hbase_test");        System.out.println(tableExists);    }}

控制台打印

20/09/13 20:49:05 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.0.171:2181,192.168.0.207:2181,192.168.0.208:2181 sessionTimeout=90000 watcher=hconnection-0x704d6e830x0, quorum=192.168.0.171:2181,192.168.0.207:2181,192.168.0.208:2181, baseZNode=/hbase20/09/13 20:49:09 INFO zookeeper.ClientCnxn: Opening socket connection to server cdh3.macro.com/192.168.0.208:2181. Will not attempt to authenticate using SASL (unknown error)20/09/13 20:49:09 INFO zookeeper.ClientCnxn: Socket connection established to cdh3.macro.com/192.168.0.208:2181, initiating session20/09/13 20:49:09 INFO zookeeper.ClientCnxn: Session establishment complete on server cdh3.macro.com/192.168.0.208:2181, sessionid = 0x10014dbded00993, negotiated timeout = 60000false20/09/13 20:49:13 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available20/09/13 20:49:17 INFO client.HBaseAdmin: Created hbase_testtrue

首次连接HBase，判断表不存在为false，创建表之后，判断表存在为true

查看HBase表，发现hbase_test表被成功创建

hbase(main):009:0> listTABLE                                                                                     Student blobstore                                                                                hbase_test                                                                               kylin_metadata40 row(s)Took 0.0595 seconds

本文大致介绍了Hbase入门需要知道的一些原理和实践，另外HBase的官方文档非常详细，是入门HBase的不二之选，推荐读者多阅读其官方文档。