Hbase与hive的区别

版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://blog.csdn.net/lds_include/article/details/88876760

Hbase与hive的区别

总的而言:

  • Hive和Hbase是两种基于Hadoop的不同技术–Hive是一种类SQL 的引擎,并且运行MapReduce 任务,Hbase 是一种在Hadoop之上的NoSQL的Key/vale数据库。当然,这两种工具是可以同时使用的。就像用Google 来搜索,用FaceBook 进行社交一样,Hive 可以用来进行统计查询,HBase 可以用来进行实时查询,数据也可以从Hive 写到Hbase,设置再从Hbase 写回Hive。

对hive而言

  1. Hive是一个构建在Hadoop 基础之上的数据仓库。通过Hive可以使用HQL语言查询存放在HDFS 上的数据。

  2. HQL是一种类SQL语言,这种语言最终被转化为Map/Reduce. 虽然Hive提供了SQL查询功能,但是Hive 不能够进行交互查询,因为它只能够在Haoop上批量的执行Hadoop。

  3. Hive 被分区为表格,表格又被进一步分割为列簇。列簇必须使用schema 定义,列簇将某一类型列集合起来(列不要求schema定义)。

  4. Hive 目前不支持更新操作。

  5. 由于hive在hadoop上运行批量操作,它需要花费很长的时间,通常是几分钟到几个小时才可以获取到查询的结果。

  6. Hive 适合用来对一段时间内的数据进行分析查询,例如,用来计算趋势或者网站的日志。

  7. Hive 不应该用来进行实时的查询。因为它需要很长时间才可以返回结果。

对HBase而言

  1. HBase 查询是通过特定的语言来编写的,这种语言需要重新学习。类SQL 的功能可以通过Apache Phonenix 实现,但这是以必须提供schema 为代价的。另外,Hbase 也并不是兼容所有的ACID 特性,虽然它支持某些特性。最后但不是最重要的–为了运行Hbase,Zookeeper是必须的,zookeeper 是一个用来进行分布式协调的服务,这些服务包括配置服务,维护元信息和命名空间服务。

  2. Hbase非常适合用来进行大数据的实时查询。Facebook用Hbase 进行消息和实时的分析。它也可以用来统计Facebook的连接数。

  3. HBase 是一种Key/Value 系统,它运行在HDFS 之上。和Hive 不一样,Hbase 的能够在

  4. 它的数据库上实时运行,而不是运行MapReduce 任务。

展开阅读全文

hivehbase整合报错

05-22

我用的是hive0.12.0 和hbase0.90.6 hadoop1.2.1rn在成功创建表后,运行select * from hbase_table limit 10;的时候报如下错误rn14/05/22 08:36:42 DEBUG parse.SemanticAnalyzer: Created Body Plan for Query Block nullrn14/05/22 08:36:42 DEBUG parse.SemanticAnalyzer: Created Plan for Query Block nullrn14/05/22 08:36:42 DEBUG parse.SemanticAnalyzer: Before logical optimizationrnTS[0]-SEL[1]-LIM[2]-FS[3]rn14/05/22 08:36:42 INFO ppd.OpProcFactory: Processing for FS(3)rn14/05/22 08:36:42 INFO ppd.OpProcFactory: Processing for LIM(2)rn14/05/22 08:36:42 INFO ppd.OpProcFactory: Processing for SEL(1)rn14/05/22 08:36:42 INFO ppd.OpProcFactory: Processing for TS(0)rn14/05/22 08:36:42 DEBUG hive.log: DDL: struct hbase_table_2 i32 key, string valuern14/05/22 08:36:42 INFO ql.Driver: rn[color=#FF0000]Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hbase.security.User.isHBaseSecurityEnabled(Lorg/apache/hadoop/conf/Configuration;)Zrn at org.apache.hadoop.hive.hbase.HBaseStorageHandler.addHBaseDelegationToken(HBaseStorageHandler.java:373)rn at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureTableJobProperties(HBaseStorageHandler.java:342)rn at org.apache.hadoop.hive.hbase.HBaseStorageHandler.configureInputJobProperties(HBaseStorageHandler.java:283)rn at org.apache.hadoop.hive.ql.plan.PlanUtils.configureJobPropertiesForStorageHandler(PlanUtils.java:755)rn at org.apache.hadoop.hive.ql.plan.PlanUtils.configureInputJobPropertiesForStorageHandler(PlanUtils.java:725)rn at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.convertToWork(SimpleFetchOptimizer.java:208)rn at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer$FetchData.access$000(SimpleFetchOptimizer.java:180)rn at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.optimize(SimpleFetchOptimizer.java:103)rn at org.apache.hadoop.hive.ql.optimizer.SimpleFetchOptimizer.transform(SimpleFetchOptimizer.java:74)rn at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:131)rn at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8392)rn at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)rn at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:441)rn at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342)rn at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977)rn at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)rn at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)rn at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)rn at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)rn at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)rn at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)rn at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)rn at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)rn at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)rn at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)rn at java.lang.reflect.Method.invoke(Method.java:606)rn at org.apache.hadoop.util.RunJar.main(RunJar.java:160)[/color]rn14/05/22 08:36:42 DEBUG fs.FileSystem: Starting clear of FileSystem cache with 2 elements.rn14/05/22 08:36:42 DEBUG fs.FileSystem: Removing filesystem for file:///rn14/05/22 08:36:42 DEBUG fs.FileSystem: Removing filesystem for file:/// 论坛

Hive启动报错-which : no hbase in

07-28

Hive配置完成后启动报错(在未启动hadoop情况下执行hive命令也是一样)rnrn下面是错误信息,希望大神能够帮忙解决。rnrn——————————————————————————————————————rnrnwhich: no hbase in (/opt/java/jdk1.8.0_101/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/hadoop/hadoop-2.6.4/bin:/opt/hadoop/hadoop-2.6.4/sbin:/root/bin:/opt/hive/apache-hive-2.1.0-bin/bin)rnSLF4J: Class path contains multiple SLF4J bindings.rnSLF4J: Found binding in [jar:file:/opt/tuniu/hive/apache-hive-2.1.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]rnSLF4J: Found binding in [jar:file:/opt/tuniu/hadoop/hadoop-2.6.4/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]rnSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.rnSLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]rnrnLogging initialized using configuration in jar:file:/opt/tuniu/hive/apache-hive-2.1.0-bin/lib/hive-common-2.1.0.jar!/hive-log4j2.properties Async: truernException in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql))rn at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:578)rn at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:518)rn at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)rn at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)rn at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)rn at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)rn at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)rn at java.lang.reflect.Method.invoke(Method.java:498)rn at org.apache.hadoop.util.RunJar.run(RunJar.java:221)rn at org.apache.hadoop.util.RunJar.main(RunJar.java:136)rnCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql))rn at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:226)rn at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:366)rn at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:310)rn at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:290)rn at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:266)rn at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:545)rn ... 9 morernCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql))rn at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3593)rn at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:236)rn at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:221)rn ... 14 morernCaused by: MetaException(message:Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql))rn at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3364)rn at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3336)rn at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3590)rn ... 16 morernrn————————————————————————————————————————————— 论坛

没有更多推荐了,返回首页