Hive集成Hbase

最新推荐文章于 2021-04-27 10:22:55 发布

hua_ed

最新推荐文章于 2021-04-27 10:22:55 发布

阅读量444

点赞数

分类专栏： hadoop

本文链接：https://blog.csdn.net/hua_ed/article/details/80626575

版权

hadoop 专栏收录该内容

35 篇文章 1 订阅

订阅专栏

在尝试集成Hive与Hbase的过程中遇到了错误，包括版本不匹配导致的NoSuchMethodError，Hiveserver连接拒绝问题，以及集成外部表时的IOException。通过升级Hive版本，停止Ambari的Hiveserver和Webhcat服务，添加Phoenix-client.jar到Hive的lib目录，最终成功实现了Hive查询Hbase。

摘要由CSDN通过智能技术生成

集成过程及遇到的问题

1.开始用ambari hive集成遇到问题，尝试用独立的hive1.2.1，但因为版本太低，异常如下，考虑继续使用ambari hive：
hive> select * from lxw1234;
OK
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.HTable.getRegionLocator()Lorg/apache/hadoop/hbase/client/RegionLocator;
        at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.setHTable(TableInputFormatBase.java:615)
        at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplitsInternal(HiveHBaseTableInputFormat.java:444)
        at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat.getSplits(HiveHBaseTableInputFormat.java:432)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:362)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:294)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:445)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:414)
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1670)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
[10.88.171.175 hdfs@mas sh]$

2.考虑继续使用ambari hive,路径在/usr/hdp/2.3.0.0-2557/hive/bin/hive；
3.启动/usr/hdp/2.3.0.0-2557/hive/bin/hive卡住，查看log，发现是hiveserver connect refused，原因是amberi启动了hiveserver和webhcat,解决方法是从ambari停掉这两个。
4.再启动,可以进入hive，查询集成hbase的外部表报错Failed with exception java.io.IOException:java.io.IOException: java.lang.reflect.InvocationTargetException
，Google https://community.hortonworks.com/questions/1652/how-can-i-query-hbase-from-hive.html 发现需要phoenix-client.jar到hive/lib下。

5.按教程http://lxw1234.com/archives/2015/06/319.htm测试，通过。

CREATE EXTERNAL TABLE abc_os_event_hb(
KEY string,
FTIME string,
ID string,
DEVICE_ID string,
EVENT_NAME string,
PACKAGE_VERSION_CODE string,
PACKAGE_VERSION_NAME string,
TIMESTAMP_ string,
COLLECT_TIME string,
CREATE_TIME string,
IS_TEST string
)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" =":key,f1:c9,f1:c1,f1:c2,f1:c3,f1:c4,f1:c5,f1:c6,f1:c7,f1:c8,f1:c10")
TBLPROPERTIES("hbase.table.name" = "abc_os_event", "hbase.mapred.output.outputtable" = "abc_os_event");