Hbase+Hive+Spark整合

已于 2022-10-31 15:08:51 修改
阅读量1k 收藏 2
点赞数 1
分类专栏： spark 文章标签： hive hbase spark
于 2021-04-07 21:33:35 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_42422698/article/details/115496420
版权

1 环境部署：
ambari
hive
hbase
spark
2 hive创建hbase映射表
（使用hbase目的是为了解决hive的数据更新弊端，做hive映射是为了使用hive -e和sparksql计算数据更便捷 ）

+----------------------------------------------------+
| CREATE EXTERNAL TABLE `ods.ods_qu_type_config_test`( |
|   `id` string COMMENT '',                          |
|   `ex_create_time` string COMMENT '',              |
|   `ex_create_user_id` string COMMENT '',           |
|   `ex_update_time` string COMMENT '',              |
|   `ex_update_user_id` string COMMENT '',           |
|   `del` string COMMENT '',                         |
|   `version` string COMMENT '',                     |
|   `name` string COMMENT '',                        |
|   `code` string COMMENT '',                        |
|   `soft_name` string COMMENT '',                   |
|   `use_soft` string COMMENT '',                    |
|   `grade_file_postfix` string COMMENT '',          |
|   `standard_file_postfix` string COMMENT '',       |
|   `is_operate` string COMMENT '',                  |
|   `offline_flag` string COMMENT '',                |
|   `teacher_flag` string COMMENT '',                |
|   `gss_support` string COMMENT '',                 |
|   `gss_support_version` string COMMENT '')         |
| ROW FORMAT SERDE                                   |
|   'org.apache.hadoop.hive.hbase.HBaseSerDe'        |
| STORED BY                                          |
|   'org.apache.hadoop.hive.hbase.HBaseStorageHandler'  |
| WITH SERDEPROPERTIES (                             |
|   'hbase.columns.mapping'='\n:key,\nCF:EX_CREATE_TIME,\nCF:EX_CREATE_USER_ID,\nCF:EX_UPDATE_TIME,\nCF:EX_UPDATE_USER_ID,\nCF:DEL,\nCF:VERSION,\nCF:NAME,\nCF:CODE,\nCF:SOFT_NAME,\nCF:USE_SOFT,\nCF:GRADE_FILE_POSTFIX,\nCF:STANDARD_FILE_POSTFIX,\nCF:IS_OPERATE,\nCF:OFFLINE_FLAG,\nCF:TEACHER_FLAG,\nCF:GSS_SUPPORT,\nCF:GSS_SUPPORT_VERSION\n',  |
|   'serialization.format'='1')                      |
| TBLPROPERTIES (                                    |
|   'bucketing_version'='2',                         |
|   'hbase.table.name'='test.qu_type_config',        |
|   'transient_lastDdlTime'='1617005582')            |
+----------------------------------------------------+

3 报错详情

21/04/07 20:28:45 ERROR Table: Unable to get field from serde: org.apache.hadoop.hive.hbase.HBaseSerDe
java.lang.RuntimeException: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found)
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:296)
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:276)
        at org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:668)
        at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:651)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:364)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:361)
        at scala.Option.map(Option.scala:146)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:361)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:359)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:278)
        at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:216)
        at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:215)
        at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:261)
        at org.apache.spark.sql.hive.client.HiveClientImpl.getTableOption(HiveClientImpl.scala:359)
        at org.apache.spark.sql.hive.client.HiveClient$class.getTable(HiveClient.scala:81)
        at org.apache.spark.sql.hive.client.HiveClientImpl.getTable(HiveClientImpl.scala:83)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
        at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTable(HiveExternalCatalog.scala:117)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
        at org.apache.spark.sql.hive.HiveExternalCatalog.getTable(HiveExternalCatalog.scala:683)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupRelation(SessionCatalog.scala:672)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:662)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.resolveRelation(Analyzer.scala:617)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:647)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:640)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:286)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:640)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:586)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)
        at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
        at scala.collection.immutable.List.foldLeft(List.scala:84)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:124)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:118)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:103)
        at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
        at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
        at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
        at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:364)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:272)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: MetaException(message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:98)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77)
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:294)
        ... 72 more
21/04/07 20:28:45 ERROR SparkSQLDriver: Failed in [select * from   ods.ods_qu_type_config_test]
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.hbase.HBaseStorageHandler
        at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:310)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:394)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:361)
        at scala.Option.map(Option.scala:146)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:361)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:359)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:278)
        at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:216)
        at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:215)
        at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:261)
        at org.apache.spark.sql.hive.client.HiveClientImpl.getTableOption(HiveClientImpl.scala:359)
        at org.apache.spark.sql.hive.client.HiveClient$class.getTable(HiveClient.scala:81)
        at org.apache.spark.sql.hive.client.HiveClientImpl.getTable(HiveClientImpl.scala:83)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
        at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTable(HiveExternalCatalog.scala:117)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
        at org.apache.spark.sql.hive.HiveExternalCatalog.getTable(HiveExternalCatalog.scala:683)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupRelation(SessionCatalog.scala:672)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:662)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.resolveRelation(Analyzer.scala:617)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:647)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:640)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:286)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:640)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:586)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)
        at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
        at scala.collection.immutable.List.foldLeft(List.scala:84)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:124)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:118)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:103)
        at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
        at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
        at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
        at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:364)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:272)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.hbase.HBaseStorageHandler
        at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:309)
        at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:305)
        ... 69 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.hbase.HBaseStorageHandler
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:304)
        ... 70 more
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.hbase.HBaseStorageHandler
        at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:310)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:394)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:361)
        at scala.Option.map(Option.scala:146)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:361)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1.apply(HiveClientImpl.scala:359)
        at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:278)
        at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:216)
        at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:215)
        at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:261)
        at org.apache.spark.sql.hive.client.HiveClientImpl.getTableOption(HiveClientImpl.scala:359)
        at org.apache.spark.sql.hive.client.HiveClient$class.getTable(HiveClient.scala:81)
        at org.apache.spark.sql.hive.client.HiveClientImpl.getTable(HiveClientImpl.scala:83)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getRawTable$1.apply(HiveExternalCatalog.scala:118)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
        at org.apache.spark.sql.hive.HiveExternalCatalog.getRawTable(HiveExternalCatalog.scala:117)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
        at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:684)
        at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
        at org.apache.spark.sql.hive.HiveExternalCatalog.getTable(HiveExternalCatalog.scala:683)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupRelation(SessionCatalog.scala:672)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:662)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.resolveRelation(Analyzer.scala:617)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:647)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:640)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:288)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:286)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:306)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
        at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:304)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:286)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:640)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:586)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:87)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:84)
        at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
        at scala.collection.immutable.List.foldLeft(List.scala:84)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:84)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:76)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:124)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:118)
        at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:103)
        at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
        at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
        at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:74)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
        at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:62)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:364)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:272)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:904)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.hbase.HBaseStorageHandler
        at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:309)
        at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:305)
        ... 69 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.hbase.HBaseStorageHandler
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:304)
        ... 70 more

5 解决办法
继续执行

spark-sql>  add jar /usr/hdp/3.0.1.0-187/hive/lib/hive-hbase-handler-3.1.0.3.0.1.0-187.jar;

继续执行，然后成功：
spark-sql> select * from ods.ods_qu_type_config_test;
6 一路上有各种其他的缺包报错
大概如下：

#必须包：
hbase-mapreduce-2.2.4.jar
hbase-shaded-miscellaneous-3.5.0.jar
hbase-shaded-protobuf-3.5.0.jar
hbase-protocol-shaded-2.2.4_1.jar
hbase-shaded-netty-3.5.0.jar
hbase-server
hbase-client
hbase-common-2

执行分发命令：
scp hbase-protocol-2.0.0.3.0.1.0-187.jar root@10.**.8.117:/usr/hdp/3.0.1.0-187/spark2/jars/

注意：
低版本的包有的是不可用的。可以用高版本代替。各jar包高版本下载地址：https://maven.aliyun.com/mvn/search

独狐游清湖
关注
  • 1
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 1
    评论
专栏目录
SparkOnHBase:SparkOnHBase
05-17
:warning: Cloudera Labs中的Spark On HBase项目在2015年被合并到上游HBase项目中，并且自CDH 5.7起CDH附带了Backport SparkOnHBase 概述 这是一个用于Spark的HBase的简单可重用库 功能性 当前功能支持以下功能 批量放置 bulkDelete bulkIncrement bulkGet bulkCheckAndPut bulkCheckAndDelete foreachPartition（带有HConnection） mapPartition（使用HConnection） hbaseRDD（HBaseInputFormat） Java API Java实例 Java单元测试 不远的将来 更多单元测试 更多例子 Python API Python示例 Python单元测试 遥远的未来 分区批量放置/突变 排序分
Spark与HBase的集成
蔡政洁的博客
12-10 4336
目录一、Spark与HBase的集成二、Spark SQL与HBase的集成 版本说明： hbase版本：hbase-1.3.1 spark版本：spark-2.4.7-bin-hadoop2.7 一、Spark与HBase的集成 背景： Spark支持多种数据源，但是Spark对HBase的读写都没有相对优雅的api，但spark和HBase整合的场景又比较多，故通过spark的数据源API自己实现了一套比较方便操作HBase的API。 数据模型： row,addres,age,username 0
Hadoop生态简介，Hive、Spark、HBase等
最新发布
随便写写，随便写写
05-16 1299
Hadoop生态全景介绍，Spark、Hive、HBase等
SparkSQL查询与HBase关联的Hive外部表，查询失败
TT-Learning
05-21 1681
1.问题 集群环境：Ambar i2.7.3 + HDP 3.0.1.0 Hive中创建一个外部表，外部表是HBase中的表，想通过SparkSQL进行查询。 在Hive中使用HQL查询，正确执行，得出正确结果（示例表是空表）。 在使用SparkSQL查询时，报错：error in initSerDe: java.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found 解决办法 拷贝以下jar包到
通过Spark访问Hbase的Hive外部表（hive on hbase的表）
TURING.DT
02-24 2667
业务处理的过程中，使用到pyspark访问hive，直接访问没有问题， 后面遇到使用pyspark访问hive关联hbase的表（hive on hbase），就出现报错了。 首先集群已经可以通过spark-sql读取hive的数据，然后进行如下配置。 1.拷贝如下jar包到${spark_home}/jars（spark2.0之前是${spark_home}/lib）: hba...
Spark与HBase的集成与数据访问
晓晓的天空
01-09 1606
通过集成Spark与HBase，可以充分利用这两个强大的工具来处理和分析大规模数据。本文深入介绍了如何集成Spark与HBase，并提供了示例代码，以帮助大家更好地理解这一过程。同时，也提供了性能优化的建议，以确保在集成过程中获得良好的性能表现。
hadoop,hbase,hive版本整合兼容性最全，最详细说明【适用于任何版本】 -
12-02
本文将详细介绍Hadoop、HBase和Hive的版本整合兼容性，以及如何确保它们在不同版本间顺畅协作。 首先，Hadoop作为基础平台，其版本选择会直接影响到HBase和Hive的运行。Hadoop的主要组件包括HDFS（分布式文件系统）...
源码主要用于学习：1. Spring Boot+Hadoop+Hive+Hbase实现数据基本操作,Hive数据源使.zip
02-24
标题中的“Spring Boot+Hadoop+Hive+Hbase实现数据基本操作”是一个关于大数据处理技术集成应用的项目。这个项目结合了四个重要的组件来处理和管理大规模数据： 1. **Spring Boot**: 是一个基于Java的框架，用于...
建立Hive和Hbase的映射关系，通过Spark将Hive表中数据导入ClickHouse
07-31
本话题关注的是如何建立Hive与HBase之间的映射关系，并利用Spark将Hive中的数据高效地导入到ClickHouse数据库。以下将详细介绍这一过程的关键步骤和涉及的技术点。 首先，Hive是基于Hadoop的数据仓库工具，用于存储...
scala-hive-HBASE-Api.7z
10-31
在Java开发中，为了将这些组件整合在一起，你需要相关的jar包，例如包含Hive和HBase的API。这些API允许你在Scala程序中直接操作Hive和HBase，简化了数据处理的流程。例如，`scalatestOne`可能是一个Scala测试项目，...
妳那伊抹微笑_云计算之Hadoop-2.2.0+Hbaase-0.96.2 +Hive-0.13.1完全分布式环境整合安装文档V1.0.0.docx
08-18
这个文档是《云计算之Flume+Kafka+Storm+Redis/Hbase+Hadoop+Hive+Mahout+Spark 技术文档分享V1.0.0》系列的一部分，涵盖了多种云计算技术。 首先，Hadoop-2.2.0是一个开源的分布式计算框架，其核心由HDFS（Hadoop ...
Class org.apache.hadoop.hive.hbase.HBaseSplit not found
weixir123的博客
11-20 619
在hive-site.xml加上 <property> <name>hive.aux.jars.path</name> <value>file:///opt/modules/hive/hive-0.9.0/lib/hive-hbase-handler-0.9.0.jar,file:///opt/modules/hive/hive-0...
spark读取HBase数据的一次坑爹经历
sgyuanshi的博客
09-29 1356
首先，在这里说明一下，我遇到的错误如下： org.apache.hadoop.hbase.DoNotRetryIOException： /192.168.x.x:16020 is unable to read call parameter from client 10.47.x.x 然后， spark读取HBase ...
Hive查询失败，连不上端口16020
花语无痕的博客
08-14 2707
执行hive查询时，报错如下： Wed Aug 14 10:44:13 CST 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=69210: Call to server31/10.45.152.31:16020 failed on connection exception: org.apac...
Hive报错：Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
weixin_34132768的博客
11-13 549
应用场景 搭建了hadoop伪分布式集群，并且在其上搭建了hive环境，hive运行报HiveMetaStoreClient错误。 解决办法 FAILED: Execution Error, return code 1 from org.apache.hadoo...
HBase&Spark集成 -- DataFrame
z1987865446的博客
04-24 3947
HBase&Spark集成 – DataFrame ​ Apache HBase 是一个成熟的 NoSQL 存储系统，已在大规模生产部署中得到证明。尽管 HBase 是一个键值存储，但对于像 SQL 一样更轻松地访问数据的需求很高。Apache Spark SQL 提供了基本过滤和插入数据的强大支持。hbase-connectors子项目提供了HBase 与SparkSQL 的集成。 ​ hbase -spark 集成 利用 Spark-1.2.0 版本中引入的DataSource API。HB
HBase与Spark的大数据处理
禅与计算机程序设计艺术
01-28 739
1.背景介绍 1. 背景介绍 大数据处理是当今计算机科学和信息技术领域的一个热门话题。随着数据量的不断增长，传统的数据处理方法已经无法满足需求。因此，大数据处理技术的研究和应用变得越来越重要。 HBase 和 Spark 是 Apache 基金会的两个开源项目，分别属于 NoSQL 数据库和大数据处理框架。HBase 是一个分布式、可扩展、高性能的列式存储系统，基于 Google 的 Big...
hmaster和hregionserver 16020 端口冲突问题
热门推荐
xishanxinyue的专栏
05-20 1万+
问题背景： 哥最近升级了下hadoop、 hbase、 spark，发现hbase升级后主节点上的hregionsever老是启动不了。 查看日志发现如下端口冲突错误：java.lang.RuntimeException: Failed construction of Regionserver: class org.apache.hadoop.hbase.regionserver.HRegion
spark操作hbase的问题汇总
12-03 683
1.无法访问hbase的regionservice 自己就在虚拟机里搭建了Hadoop和HBase，在进行API操作连接时，出现以下错误： Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions: Fri N...
hbase+hive+spark整合
06-06
HBase、Hive和Spark是三个不同的开源大数据技术，它们可以被整合在一起，以实现更高效的数据处理和分析。具体来说，HBase是一个分布式的NoSQL数据库，可以存储海量的结构化数据；Hive是一个数据仓库工具，可以将结构化数据转换为SQL查询；而Spark是一个快速的分布式计算引擎，可以对大规模数据进行实时处理和分析。通过将这三个技术整合在一起，可以实现更高效的数据存储、查询和分析，从而提高数据处理的效率和准确性。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

独狐游清湖

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值