Hbase中常用shell命令以及命令在java中的使用，Mappreduce集成Hbase

最新推荐文章于 2023-06-05 00:15:00 发布

1只筷子

最新推荐文章于 2023-06-05 00:15:00 发布

阅读量314

点赞数

分类专栏： habse hadoop 文章标签： hbase hdfs hadoop 大数据

本文链接：https://blog.csdn.net/qq_42346966/article/details/103898435

版权

habse 同时被 2 个专栏收录

4 篇文章 0 订阅

订阅专栏

hadoop

3 篇文章 0 订阅

订阅专栏

Hbase安装（伪分布）

HDFS基本环境(存储)

1,安装JDK,配置环境变量JAVA_HOME

   [root@CentOS ~]# rpm -ivh jdk-8u171-linux-x64.rpm

Preparing...                          ################################# [100%]
Updating / installing...
   1:jdk1.8-2000:1.8.0_171-fcs        ################################# [100%]
Unpacking JAR files...
        tools.jar...
        plugin.jar...
        javaws.jar...
        deploy.jar...
        rt.jar...
        jsse.jar...
        charsets.jar...
        localedata.jar...

  [root@CentOS ~]# vi .bashrc

JAVA_HOME=/usr/java/latest
CLASSPATH=.
PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME
export CLASSPATH
export PATH

[root@CentOS ~]# source .bashrc 
[root@CentOS ~]# jps
1933 Jps

2,关闭防火墙

 [root@CentOS ~]# systemctl stop firewalld # 关闭 服务
    [root@CentOS ~]# systemctl disable firewalld # 关闭开机自启动
    Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
    Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
    [root@CentOS ~]# firewall-cmd --state
    not running

3,配置主机名和IP映射关系

[root@CentOS ~]# cat /etc/hostname 
CentOS
[root@CentOS ~]# vi /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.186.150 CentOS

4,配置SSH免密码登录

[root@CentOS ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:6yYiypvclJAZLU2WHvzakxv6uNpsqpwk8kzsjLv3yJA root@CentOS
The key's randomart image is:
+---[RSA 2048]----+
|  .o.            |
|  =+             |
| o.oo            |
|  =. .           |
| +  o . S        |
| o...=   .       |
|E.oo. + .        |
|BXX+o....        |
|B#%O+o o.        |
+----[SHA256]-----+
[root@CentOS ~]# ssh-copy-id CentOS
[root@CentOS ~]# ssh CentOS
Last failed login: Mon Jan  6 14:30:49 CST 2020 from centos on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Mon Jan  6 14:20:27 2020 from 192.168.186.1

5,上传Hadoop安装包,并解压到/usr目录

[root@CentOS ~]# tar -zxf  hadoop-2.9.2.tar.gz -C /usr/

6,配置HADOOP_HOME环境变量

[root@CentOS ~]# vi .bashrc

 HADOOP_HOME=/usr/hadoop-2.9.2
    JAVA_HOME=/usr/java/latest
    CLASSPATH=.
    PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
    export JAVA_HOME
    export CLASSPATH
    export PATH
    export HADOOP_HOME

[root@CentOS ~]# source .bashrc    
[root@CentOS ~]# hadoop classpath #打印Hadoop的类路径
/usr/hadoop-2.9.2/etc/hadoop:/usr/hadoop-2.9.2/share/hadoop/common/lib/*:/usr/hadoop-2.9.2/share/hadoop/common/*:/usr/hadoop-2.9.2/share/hadoop/hdfs:/usr/hadoop-2.9.2/share/hadoop/hdfs/lib/*:/usr/hadoop-2.9.2/share/hadoop/hdfs/*:/usr/hadoop-2.9.2/share/hadoop/yarn:/usr/hadoop-2.9.2/share/hadoop/yarn/lib/*:/usr/hadoop-2.9.2/share/hadoop/yarn/*:/usr/hadoop-2.9.2/share/hadoop/mapreduce/lib/*:/usr/hadoop-2.9.2/share/hadoop/mapreduce/*:/usr/hadoop-2.9.2/contrib/capacity-scheduler/*.jar

7,修改core-site.xml

[root@CentOS ~]# vi /usr/hadoop-2.9.2/etc/hadoop/core-site.xml

  <!--nn访问入口-->
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://CentOS:9000</value>
    </property>
    <!--hdfs工作基础目录-->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/hadoop-2.9.2/hadoop-${user.name}</value>
    </property>

8,修改hdfs-site.xml

[root@CentOS ~]# vi /usr/hadoop-2.9.2/etc/hadoop/hdfs-site.xml

   <!--block副本因子-->
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <!--配置Sencondary namenode所在物理主机-->
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>CentOS:50090</value>
    </property>
    <!--设置datanode最大文件操作数-->
    <property>
            <name>dfs.datanode.max.xcievers</name>
            <value>4096</value>
    </property>
    <!--设置datanode并行处理能力-->
    <property>
            <name>dfs.datanode.handler.count</name>
            <value>6</value>
    </property>

9,修改slaves

[root@CentOS ~]# vi /usr/hadoop-2.9.2/etc/hadoop/slaves 

CentOS

10,格式化NameNode,生成fsimage

[root@CentOS ~]# hdfs namenode -format
[root@CentOS ~]# yum install -y tree
[root@CentOS ~]# tree /usr/hadoop-2.9.2/hadoop-root/
/usr/hadoop-2.9.2/hadoop-root/
└── dfs
    └── name
        └── current
            ├── fsimage_0000000000000000000
            ├── fsimage_0000000000000000000.md5
            ├── seen_txid
            └── VERSION

3 directories, 4 files

11,启动HDFS服务

[root@CentOS ~]# start-dfs.sh

Zookeeper安装(协调)

1,上传zookeeper的安装包,并解压在/usr目录下

[root@CentOS ~]# tar -zxf zookeeper-3.4.12.tar.gz -C /usr/

2,配置Zookepeer的zoo.cfg

[root@CentOS ~]# tar -zxf zookeeper-3.4.12.tar.gz -C /usr/
[root@CentOS ~]# cd /usr/zookeeper-3.4.12/
[root@CentOS zookeeper-3.4.12]# cp conf/zoo_sample.cfg conf/zoo.cfg
[root@CentOS zookeeper-3.4.12]# vi conf/zoo.cfg 

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/root/zkdata
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

3,创建zookeeper的数据目录

[root@CentOS ~]# mkdir /root/zkdata

4,启动zookeeper服务

[root@CentOS ~]# cd /usr/zookeeper-3.4.12/
[root@CentOS zookeeper-3.4.12]# ./bin/zkServer.sh start zoo.cfg
ZooKeeper JMX enabled by default
Using config: /usr/zookeeper-3.4.12/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@CentOS zookeeper-3.4.12]# ./bin/zkServer.sh status zoo.cfg
ZooKeeper JMX enabled by default
Using config: /usr/zookeeper-3.4.12/bin/../conf/zoo.cfg
Mode: standalone

Hbase配置与安装(数据库服务)

1,上传Hbase安装包,并解压到/usr目录下

[root@CentOS ~]# tar -zxf hbase-1.2.4-bin.tar.gz -C /usr/

2,配置Hbase环境变量HBASE_HOME

[root@CentOS ~]# vi .bashrc

HBASE_HOME=/usr/hbase-1.2.4
HADOOP_HOME=/usr/hadoop-2.9.2
JAVA_HOME=/usr/java/latest
CLASSPATH=.
PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin
export JAVA_HOME
export CLASSPATH
export PATH
export HADOOP_HOME
export HBASE_HOME

[root@CentOS ~]# source .bashrc 
[root@CentOS ~]# hbase classpath # 测试Hbase是否识别Hadoop
/usr/hbase-1.2.4/conf:/usr/java/latest/lib/tools.jar:/usr/hbase-1.2.4:/usr/hbase-1.2.4/lib/activation-1.1.jar:/usr/hbase-1.2.4/lib/aopalliance-1.0.jar:/usr/hbase-1.2.4/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hbase-1.2.4/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hbase-1.2.4/lib/api-asn1-api-1.0.0-M20.jar:/usr/hbase-1.2.4/lib/api-util-1.0.0-M20.jar:/usr/hbase-1.2.4/lib/asm-3.1.jar:/usr/hbase-1.2.4/lib/avro-
...
1.7.4.jar:/usr/hbase-1.2.4/lib/commons-beanutils-1.7.0.jar:/usr/hbase-1.2.4/lib/commons-
2.9.2/share/hadoop/yarn/*:/usr/hadoop-2.9.2/share/hadoop/mapreduce/lib/*:/usr/hadoop-2.9.2/share/hadoop/mapreduce/*:/usr/hadoop-2.9.2/contrib/capacity-scheduler/*.jar

3,配置hbase-site.xml

[root@CentOS ~]# cd /usr/hbase-1.2.4/
[root@CentOS hbase-1.2.4]# vi conf/hbase-site.xml

   <property>
        <name>hbase.rootdir</name>
        <value>hdfs://CentOS:9000/hbase</value>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>CentOS</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
    </property>

4,修改hbase-env.sh,将HBASE_MANAGES_ZK修改为false

[root@CentOS ~]# cd /usr/hbase-1.2.4/
[root@CentOS hbase-1.2.4]# grep -i HBASE_MANAGES_ZK conf/hbase-env.sh 
# export HBASE_MANAGES_ZK=true
[root@CentOS hbase-1.2.4]# vi conf/hbase-env.sh 
export HBASE_MANAGES_ZK=false
[root@CentOS hbase-1.2.4]# grep -i HBASE_MANAGES_ZK conf/hbase-env.sh 
export HBASE_MANAGES_ZK=false

export HBASE_MANAGES_ZK=false告知Hbase,使用外部Zookeeper

5,启动Hbase

[root@CentOS hbase-1.2.4]# ./bin/start-hbase.sh 
starting master, logging to /usr/hbase-1.2.4/logs/hbase-root-master-CentOS.out
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
starting regionserver, logging to /usr/hbase-1.2.4/logs/hbase-root-1-regionserver-CentOS.out
[root@CentOS hbase-1.2.4]# jps
3090 NameNode
5027 HMaster
3188 DataNode
5158 HRegionServer
3354 SecondaryNameNode
5274 Jps
3949 QuorumPeerMain

6,验证Hbase安装是否成功

WebUI验证 http://192.168.186.150:16010/
Hbase shell验证(靠谱)

Shell 命令-掌握

###基本命令

1、打开Hbase Shell

[root@CentOS hbase-1.2.4]# ./bin/hbase shell

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hbase-1.2.4/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.4, rUnknown, Wed Feb 15 18:58:00 CST 2017

2，查看服务器状态

 hbase(main):001:0> status

1 active master, 0 backup masters, 1 servers, 0 dead, 2.0000 average load

3，查看当前版本信息

   hbase(main):002:0> version

1.2.4, rUnknown, Wed Feb 15 18:58:00 CST 2017

hbase(main):003:0>

###Namespace-数据库

alter_namespace, create_namespace, describe_namespace, drop_namespace, list_namespace, list_namespace_tables

1、创建namespace

 hbase(main):007:0> create_namespace 'baizhi',{'user'=>'jiangzz'}
    0 row(s) in 0.0390 seconds

2、查看namespace详情

hbase(main):009:0> describe_namespace 'baizhi'
DESCRIPTION                                                                     
{NAME => 'baizhi', user => 'jiangzz'}                            
1 row(s) in 0.0110 seconds

3、修改namespace

  hbase(main):008:0>  alter_namespace 'baizhi', {METHOD => 'set', 'sex' => 'true'}
    0 row(s) in 0.0350 seconds
    
    hbase(main):010:0> describe_namespace 'baizhi'
    DESCRIPTION                                                                     
    {NAME => 'baizhi', sex => 'true', user => 'jiangzz'}                            
    1 row(s) in 0.0080 seconds
    
    hbase(main):011:0>  alter_namespace 'baizhi',{METHOD => 'unset', NAME=>'sex'}
    0 row(s) in 0.1140 seconds
    
    hbase(main):012:0> describe_namespace 'baizhi'
    DESCRIPTION                                                                     
    {NAME => 'baizhi', user => 'jiangzz'}                                           
    1 row(s) in 0.0090 seconds

4、查看所有的namespace

 hbase(main):013:0> list_namespace
    NAMESPACE                                                                       
    baizhi                                                                          
    default                                                                         
    hbase                                                                           
    3 row(s) in 0.0270 seconds
    
    hbase(main):020:0>  list_namespace '^b.*'
    NAMESPACE                                                                       
    baizhi                                                                          
    1 row(s) in 0.0140 seconds

5、查看namespace的表

hbase(main):021:0> list_namespace_tables 'hbase'
TABLE                                                                           
meta                                                                            
namespace                                                                       
2 row(s) in 0.0210 seconds

6、删除namespace


```java
   hbase(main):022:0> drop_namespace 'baizhi'
    0 row(s) in 0.0260 seconds
    
    hbase(main):023:0> list_namespace
    NAMESPACE                                                                       
    default                                                                         
    hbase                                                                           
    2 row(s) in 0.0100 seconds


## 表的操作

1、 查看所有表-(用户表)


```yaml
hbase(main):004:0> list

    TABLE                                                                           
    0 row(s) in 0.0350 seconds

2、创建表

hbase(main):045:0> create baizhi:t_user',`{NAME=>'cf1',VERSIONS=>3,BLOCKCACHE => true},{NAME=>'cf2',TTL=>300}
0 row(s) in 2.3100 seconds

=> Hbase::Table - baizhi:t_user

VERSIONS:保留数据版本,默认值1 TTL:列簇下列存活时间,默认是FOREVER BLOCKCACHE:是否开启缓存,用于加快读.IN_MEMORY:设置是否将列簇下所有数据加载内存中,加速读写,默认值false,BLOOMFILTER:配置布隆过滤器(一种数据文件过滤机制),默认值ROW,可选值两个ROW|ROWCOL,如果修改为ROWCOL系统需要额外开销存储列信息作为过滤文件的索引.

3、查看table详情

 hbase(main):001:0> desc 'baizhi:t_user'
    Table baizhi:t_user is ENABLED                                                                                        
    baizhi:t_user                                                                                                         
    COLUMN FAMILIES DESCRIPTION                                                                                           
    {NAME => 'cf1', BLOOMFILTER => 'ROW', VERSIONS => '3', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK
    _ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => 
    '65536', REPLICATION_SCOPE => '0'}                                                                                    
    {NAME => 'cf2', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK
    _ENCODING => 'NONE', TTL => '300 SECONDS (5 MINUTES)', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true
    ', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}                                                                    
    2 row(s) in 0.9210 seconds

4、禁用表

  hbase(main):027:0> disable_all 'baizhi:t_u.*'
    baizhi:t_user                                                                   
    
    Disable the above 1 tables (y/n)?
    y
    1 tables successfully disabled

5、启用表

 hbase(main):028:0> enable_all 'baizhi:t_u.*'
    baizhi:t_user                                                                   
    
    Enable the above 1 tables (y/n)?
    y
    1 tables successfully enabled

6、截断表

 hbase(main):034:0> truncate 'baizhi:t_user'
    Truncating 'baizhi:t_user' table (it may take a while):
     - Disabling table...
     - Truncating table...
    0 row(s) in 3.3620 seconds

7、删除表

  hbase(main):035:0> drop 'baizhi:t_user'
    
    ERROR: Table baizhi:t_user is enabled. Disable it first.
    
    hbase(main):037:0> disable 'baizhi:t_user'
    0 row(s) in 2.2450 seconds
    
    hbase(main):038:0> drop 'baizhi:t_user'
    0 row(s) in 1.2590 seconds

DML操作

1、put添加

hbase(main):004:0> put 'baizhi:t_user','001','cf1:name','zhangsan'
0 row(s) in 0.7430 seconds

hbase(main):005:0> put 'baizhi:t_user','001','cf1:age',18
0 row(s) in 0.1120 seconds

hbase(main):006:0> put 'baizhi:t_user','001','cf1:age',20 
0 row(s) in 0.0720 seconds

2、get查一个

 hbase(main):008:0> get 'baizhi:t_user','001'
    COLUMN                              CELL
     cf1:age                            timestamp=1553961219305, value=20
     cf1:name                           timestamp=1553961181804, value=zhangsan
    
    hbase(main):009:0> get 'baizhi:t_user','001',{COLUMN=>'cf1',VERSIONS=>3}
    COLUMN                              CELL
     cf1:age                            timestamp=1553961219305, value=20
     cf1:age                            timestamp=1553961198084, value=18
     cf1:name                           timestamp=1553961181804, value=zhangsan
    3 row(s) in 0.1540 seconds
    
    hbase(main):010:0> get 'baizhi:t_user','001',{COLUMN=>'cf1',TIMESTAMP=>1553961198084}
    COLUMN                              CELL
     cf1:age                            timestamp=1553961198084, value=18
    1 row(s) in 0.0900 seconds
    
    hbase(main):015:0> get 'baizhi:t_user','001',{COLUMN=>'cf1',TIMERANGE=>[1553961198084,1553961219306],VERSIONS=>3}
    COLUMN                              CELL
     cf1:age                            timestamp=1553961219305, value=20
     cf1:age                            timestamp=1553961198084, value=18
    2 row(s) in 0.0180 seconds
    
    hbase(main):018:0> get 'baizhi:t_user','001',{COLUMN=>'cf1',FILTER => "ValueFilter(=, 'binary:zhangsan')"}
    COLUMN                              CELL
     cf1:name                           timestamp=1553961181804, value=zhangsan
    1 row(s) in 0.0550 seconds
    
    hbase(main):019:0> get 'baizhi:t_user','001',{COLUMN=>'cf1',FILTER => "ValueFilter(=, 'substring:zhang')"}
    COLUMN                              CELL
     cf1:name                           timestamp=1553961181804, value=zhangsan
    1 row(s) in 0.0780 seconds

3、delete/deleteal

 # 删除指定版本之前的所以cell
    hbase(main):027:0> delete 'baizhi:t_user','001','cf1:age',1553961899630
    0 row(s) in 0.1020 seconds
    # 删除cf1：age的所有单元格
    hbase(main):031:0> delete 'baizhi:t_user','001','cf1:age'
    0 row(s) in 0.0180 seconds
    
    hbase(main):034:0> deleteall 'baizhi:t_user','001'
    0 row(s) in 0.0360 seconds
    
    hbase(main):035:0> t.count
    0 row(s) in 0.0450 seconds
    => 0
    hbase(main):036:0> get 'baizhi:t_user','001',{COLUMN=>'cf1',VERSIONS=>3}
    COLUMN                              CELL
    0 row(s) in 0.0130 seconds

4、scan

  hbase(main):045:0> scan 'baizhi:t_user'
    ROW                                 COLUMN+CELL
     001                                column=cf1:age, timestamp=1553962118964, value=21
     001                                column=cf1:name, timestamp=1553962147916, value=zs
     002                                column=cf1:age, timestamp=1553962166894, value=19
     002                                column=cf1:name, timestamp=1553962157743, value=ls
     003                                column=cf1:name, timestamp=1553962203754, value=zl
     005                                column=cf1:age, timestamp=1553962179379, value=19
     005                                column=cf1:name, timestamp=1553962192054, value=ww
    
    hbase(main):054:0> scan 'baizhi:t_user',{ LIMIT => 2,STARTROW=>"003",REVERSED=>true}
    ROW                                 COLUMN+CELL
     003                                column=cf1:name, timestamp=1553962203754, value=zl
     002                                column=cf1:age, timestamp=1553962166894, value=19
     002                                column=cf1:name, timestamp=1553962157743, value=ls
    
    hbase(main):058:0> scan 'baizhi:t_user',{ LIMIT => 2,STARTROW=>"003",REVERSED=>true,VERSIONS=>3,TIMERANGE=>[1553962157743,1553962203790]}
    ROW                                 COLUMN+CELL
     003                                column=cf1:name, timestamp=1553962203754, value=zl
     002                                column=cf1:age, timestamp=1553962166894, value=19
     002                                column=cf1:name, timestamp=1553962157743, value=ls
    2 row(s) in 0.0810 seconds

Hbase Java API

##导入Hbase开发相关依赖

  <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-client</artifactId>
        <version>1.2.4</version>
    </dependency>

创建Admin&Conn

   private Admin admin;//负责对数据库实行 DDL
    private Connection  conn;// 表示链接,是访问Hbase入口
    
    @Before
    public void before() throws IOException {
        Configuration conf= HBaseConfiguration.create();
        conf.set(HConstants.ZOOKEEPER_QUORUM,"CentOS");
        conn= ConnectionFactory.createConnection(conf);
        admin=conn.getAdmin();
    }
    @After
    public void after() throws IOException {
        admin.close();
        conn.close();
    }

Namespace操作

获取所有namespace

 @Test
    public void testListNamespace() throws IOException {
        NamespaceDescriptor[] descriptors = admin.listNamespaceDescriptors();
        for (int i = 0; i < descriptors.length; i++) {
            NamespaceDescriptor descriptor = descriptors[i];
            System.out.println(descriptor.getName());
        }
    }

创建Namespace

@Test
public void testCreateNamespace() throws IOException {
    NamespaceDescriptor nd=NamespaceDescriptor.create("zpark")
        .addConfiguration("author","jiangzz")
        .build();
    admin.createNamespace(nd);
}

修改namespace

 @Test
    public void testModifyNamespace() throws IOException {
        NamespaceDescriptor nd=NamespaceDescriptor.create("zpark")
            .addConfiguration("key","lisi")
            .removeConfiguration("author")
            .build();
        admin.modifyNamespace(nd);
    }

删除Namespace

  @Test
    public void testDropNamespace() throws IOException {
        admin.deleteNamespace("zpark");
    }

仅仅只能删除空namespace

表操作

查看所有表

@Test
public void testListTables() throws IOException {
    HTableDescriptor[] tableDescriptors = admin.listTables();
    for (int i = 0; i < tableDescriptors.length; i++) {
        HTableDescriptor descriptor = tableDescriptors[i];
        System.out.println(descriptor);
    }
}
@Test
public void testListNamespaceWithTable() throws IOException {
    HTableDescriptor[] tableDescriptors = admin.listTableDescriptorsByNamespace("hbase");
    for (int i = 0; i < tableDescriptors.length; i++) {
        HTableDescriptor descriptor = tableDescriptors[i];
        System.out.println(descriptor);
    }
}

创建表

 @Test
    public void testCreateTable() throws IOException {
        TableName tName=TableName.valueOf("baizhi:t_user");
        HTableDescriptor tableDescriptor=new HTableDescriptor(tName);
    
        HColumnDescriptor cf1=new HColumnDescriptor("cf1");
        cf1.setMaxVersions(3);
        cf1.setInMemory(true);
        cf1.setBloomFilterType(BloomType.ROWCOL);
    
        HColumnDescriptor cf2=new HColumnDescriptor("cf2");
        cf2.setMaxVersions(3);
        cf2.setTimeToLive(300);
        cf2.setBlockCacheEnabled(true);
    
        tableDescriptor.addFamily(cf1);
        tableDescriptor.addFamily(cf2);
    
        admin.createTable(tableDescriptor);
    }

删除表

 @Test
    public void dropTable() throws IOException {
        TableName tName=TableName.valueOf("baizhi:t_user");
        if(!admin.isTableDisabled(tName)){
            admin.disableTable(tName);
        }
        admin.deleteTable(tName);
    }

CRUD(重点)

不带缓冲区

 Table table = conn.getTable(TableName.valueOf("baizhi:t_user"));
    String[] depts=new String[]{"研发部","销售部","行政部","财务部","后勤部"};
    
    for(Integer i=0;i<100;i++){
        DecimalFormat decimalFormat = new DecimalFormat("000");
        String rowKey=decimalFormat.format(i);
        Put put = new Put(rowKey.getBytes());
        put.addColumn("cf1".getBytes(),"name".getBytes(),("user"+rowKey).getBytes());
        put.addColumn("cf1".getBytes(),"age".getBytes(),(i+"").getBytes());
        put.addColumn("cf1".getBytes(),"salary".getBytes(),(i*100+"").getBytes());
        put.addColumn("cf1".getBytes(),"dept".getBytes(),(depts[i%5]).getBytes());
    
        table.put(put);
    }
    
    table.close();

批量写入

  Table table = conn.getTable(TableName.valueOf("baizhi:t_user"));
    String[] depts=new String[]{"研发部","销售部","行政部","财务部","后勤部"};
    BufferedMutator bufferedMutator=conn.getBufferedMutator(TableName.valueOf("baizhi:t_user"));
    for(Integer i=100;i<200;i++){
        DecimalFormat decimalFormat = new DecimalFormat("000");
        String rowKey=decimalFormat.format(i);
        Put put = new Put(rowKey.getBytes());
        put.addColumn("cf1".getBytes(),"name".getBytes(),("user"+rowKey).getBytes());
        put.addColumn("cf1".getBytes(),"age".getBytes(),(i+"").getBytes());
        put.addColumn("cf1".getBytes(),"salary".getBytes(),(i*100+"").getBytes());
        put.addColumn("cf1".getBytes(),"dept".getBytes(),(depts[i%5]).getBytes());
    
        bufferedMutator.mutate(put);
        if(i%50==0){
            bufferedMutator.flush();
        }
    }
    bufferedMutator.close();

指定过滤条件

 @Test
    public void testGetOneRow01() throws IOException {
        Table table = conn.getTable(TableName.valueOf("baizhi:t_user"));
        String rowKey="000";
        //构建单个查询条件-重点
        Get get=new Get(rowKey.getBytes());
        get.setMaxVersions(3);//获取最新三个版本
        //get.setTimeStamp(1578330481699L);//指定时间戳查询
        get.setTimeRange(1578330481699L,1578345936177L);//查询区间,不包含最大时间前闭后开
        Filter filter1=new QualifierFilter(CompareFilter.CompareOp.EQUAL,new SubstringComparator("ge"));
        Filter filter2=new QualifierFilter(CompareFilter.CompareOp.EQUAL,new BinaryComparator("name".getBytes()));
        Filter list=new FilterList(FilterList.Operator.MUST_PASS_ONE,filter1,filter2);
        get.setFilter(list);//对cell进行过滤
    
        //遍历结果-重点
        Result result = table.get(get);
    
        CellScanner cellScanner = result.cellScanner();
        while(cellScanner.advance()){
            Cell cell = cellScanner.current();
            String family = Bytes.toString(CellUtil.cloneFamily(cell));
            String qualifier = Bytes.toString(CellUtil.cloneQualifier(cell));
            String value = Bytes.toString(CellUtil.cloneValue(cell));
            long version= cell.getTimestamp();
            String key=Bytes.toString(cell.getRow());
            System.out.println(key+"\t"+family+":"+qualifier+"\t"+value+" "+version);
        }
    
        table.close();
    }

查询最新记录

@Test
public void testGetOneRow02() throws IOException {
    Table table = conn.getTable(TableName.valueOf("baizhi:t_user"));
    String rowKey="002";
    Get get=new Get(rowKey.getBytes());
    //遍历结果-重点
    Result result = table.get(get);

    String key = Bytes.toString(result.getRow());
    String name = Bytes.toString(result.getValue("cf1".getBytes(),"name".getBytes()));
    String age =  Bytes.toString(result.getValue("cf1".getBytes(),"age".getBytes()));
    String salary = Bytes.toString(result.getValue("cf1".getBytes(),"salary".getBytes()));
    String dept = Bytes.toString(result.getValue("cf1".getBytes(),"dept".getBytes()));
    System.out.println(key+"\t"+name+","+age+","+salary+","+dept);


    table.close();
}

获取某个单元格的所有版本

 @Test
    public void testGetOneRow03() throws IOException {
        Table table = conn.getTable(TableName.valueOf("baizhi:t_user"));
        String rowKey="000";
        Get get=new Get(rowKey.getBytes());
        get.setMaxVersions();
        //遍历结果-重点
        Result result = table.get(get);
        List<Cell> nameCells = result.getColumnCells("cf1".getBytes(), "name".getBytes());
        for (Cell nameCell : nameCells) {
            System.out.println(Bytes.toString(CellUtil.cloneValue(nameCell)));
        }
        table.close();
    }

DELETE|DELETEALL

删除所有单元

 @Test
    public void testDeleteAll() throws IOException {
        Table table = conn.getTable(TableName.valueOf("baizhi:t_user"));
        String rowKey="001";
        Delete delete=new Delete(rowKey.getBytes());
        table.delete(delete);
        table.close();
    }

删除指定版本数据

@Test
public void testDeleteOneCell() throws IOException {
Table table = conn.getTable(TableName.valueOf(“baizhi:t_user”));
String rowKey=“000”;
Delete delete=new Delete(rowKey.getBytes());
delete.addColumn(“cf1”.getBytes(),“name”.getBytes(),1578330481699L);
table.delete(delete);
table.close();
}

SCAN

  @Test
    public void testScan01() throws IOException {
        Table table = conn.getTable(TableName.valueOf("baizhi:t_user"));
    
        Scan scan = new Scan();
        //        scan.addFamily("cf1".getBytes());//指定检索列簇
        //        scan.setStartRow("010".getBytes());
        //        scan.setStopRow("020".getBytes());
        //        scan.setTimeRange(157830000000L,1578350584987L);
        Filter filter1=new FuzzyRowFilter(Arrays.asList(new Pair<byte[], byte[]>("000".getBytes(),new byte[]{0,0,1})));
        //new RowFilter(CompareFilter.CompareOp.GREATER,new BinaryComparator("190".getBytes()));
        //new PrefixFilter("01".getBytes());
        Filter filter2=new RowFilter(CompareFilter.CompareOp.GREATER,new BinaryComparator("005".getBytes()));
    
        Filter list=new FilterList(FilterList.Operator.MUST_PASS_ALL,filter1,filter2);
        scan.setFilter(list);
    
        ResultScanner resultScanner = table.getScanner(scan);
        Iterator<Result> resultIterator = resultScanner.iterator();
        while(resultIterator.hasNext()){
            Result result = resultIterator.next();
            String key = Bytes.toString(result.getRow());
            String name = Bytes.toString(result.getValue("cf1".getBytes(),"name".getBytes()));
            String age =  Bytes.toString(result.getValue("cf1".getBytes(),"age".getBytes()));
            String salary = Bytes.toString(result.getValue("cf1".getBytes(),"salary".getBytes()));
            String dept = Bytes.toString(result.getValue("cf1".getBytes(),"dept".getBytes()));
            System.out.println(key+"\t"+name+"\t"+age+"\t"+salary+"\t"+dept);
        }
    
        table.close();
    }

Mappreduce集成Hbase （重点）

一，引入依赖

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.9.2</version>
</dependency>

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>2.9.2</version>
</dependency>

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
    <version>2.9.2</version>
</dependency>

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-core</artifactId>
    <version>2.9.2</version>
</dependency>
<!--Hbase依赖-->
<dependency>
    <groupId>org.apache.hbase</groupId>
    <artifactId>hbase-client</artifactId>
    <version>1.2.4</version>
</dependency>

<dependency>
    <groupId>org.apache.hbase</groupId>
    <artifactId>hbase-server</artifactId>
    <version>1.2.4</version>
</dependency>

- 配置YARN计算

1，配置yarn-site.xml

<!--配置MapReduce计算框架的核心实现Shuffle-洗牌-->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<!--配置资源管理器所在的目标主机-->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>CentOS</value>
</property>

2、配置mapred-site.xml

   <!--MapRedcue框架资源管理器的实现-->
    <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
    </property>

3、启动yarn计算

[root@CentOS ~]# start-yarn.sh

4、配置HADOOP_ClASSPATH

 HBASE_HOME=/usr/hbase-1.2.4
    HADOOP_HOME=/usr/hadoop-2.9.2
    JAVA_HOME=/usr/java/latest
    CLASSPATH=.
    PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin
    export JAVA_HOME
    export CLASSPATH
    export PATH
    export HADOOP_HOME
    export HBASE_HOME
    export HADOOP_CLASSPATH=`hbase classpath`

5、提交任务

[root@CentOS ~]# hadoop jar xxxx.jar 入口类

Job类

  public class MapReduceTotalSalaryCount extends Configured implements Tool {
    
        public int run(String[] args) throws Exception {
            //1.构建Job对象
            Configuration conf=getConf();
            conf.set(HConstants.ZOOKEEPER_QUORUM,"CentOS");
            Job job=Job.getInstance(conf,"MapReduceTotalSalaryCount");
            job.setJarByClass(MapReduceTotalSalaryCount.class);
    
            TableMapReduceUtil.initTableMapperJob(
                "baizhi:t_user",
                new Scan(),
                UserMapper.class,
                Text.class,
                DoubleWritable.class,
                job
            );
            // create 'baizhi:t_result','cf1'
            TableMapReduceUtil.initTableReducerJob(
                "baizhi:t_result",
                UserReducer.class,
                job
            );
    
            //提交Job作业
            job.waitForCompletion(true);
            return 0;
        }
        public static void main(String[] args) throws Exception {
            ToolRunner.run(new MapReduceTotalSalaryCount(),args);
        }
    }

Map类

 public class UserMapper extends TableMapper<Text, DoubleWritable> {
        @Override
        protected void map(ImmutableBytesWritable key, Result value, Context context) throws IOException, InterruptedException {
            String dept = Bytes.toString(value.getValue("cf1".getBytes(), "dept".getBytes()));
            String salary = Bytes.toString(value.getValue("cf1".getBytes(), "salary".getBytes()));
            context.write(new Text(dept),new DoubleWritable(Double.parseDouble(salary)));
        }
    }

Reduce类

  public class UserReducer extends TableReducer<Text,DoubleWritable,Text> {
    
        @Override
        protected void reduce(Text key, Iterable<DoubleWritable> values, Context context) throws IOException, InterruptedException {
            Double total=0.0;
            for (DoubleWritable value : values) {
                total += value.get();
            }
            Put put = new Put(key.toString().getBytes());
            put.addColumn("cf1".getBytes(),"sum".getBytes(),(total+"").getBytes());
            context.write(null,put);
        }
    }

Combiner类

  public class UserCombiner extends Reducer<Text, DoubleWritable, Text,DoubleWritable> {
        @Override
        protected void reduce(Text key, Iterable<DoubleWritable> values, Context context) throws IOException, InterruptedException {
            Double total=0.0;
            for (DoubleWritable value : values) {
                total += value.get();
            }
            context.write(key,new DoubleWritable(total));
        }
    }

除此之外还可以集成hdfs,和windos本地

1只筷子

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Hbase中常用shell命令以及命令在java中的使用，Mappreduce集成Hbase

Hbase安装（伪分布）HDFS基本环境(存储)1,安装JDK,配置环境变量JAVA_HOME [root@CentOS ~]# rpm -ivh jdk-8u171-linux-x64.rpm Preparing... ################################# [100%]Updating / insta...
复制链接

扫一扫

专栏目录