spark sql 访问hive数据时找不mysql的解决方法

最新推荐文章于 2021-07-19 15:59:54 发布

JokerCao

最新推荐文章于 2021-07-19 15:59:54 发布

阅读量2.7k

点赞数

分类专栏： spark 文章标签： spark sql hive mysql驱动 classpath

本文链接：https://blog.csdn.net/sunflower_cao/article/details/38561639

版权

当使用Spark SQL访问Hive数据时，遇到找不到MySQL驱动的错误。解决办法是在启动Spark时通过--driver-class-path参数添加MySQL驱动到类路径中。

摘要由CSDN通过智能技术生成

1现象：

错误如下：

org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table SOGOUQ1
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:958)
        at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:924)
        at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:54)
        at org.apache.spark.sql.hive.HiveContext$anon$1.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$super$lookupRelation(HiveContext.scala:143)
        at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$anonfun$lookupRelation$3.apply(Catalog.scala:96)
        at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$anonfun$lookupRelation$3.apply(Catalog.scala:96)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation(Catalog.scala:96)
        at org.apache.spark.sql.hive.HiveContext$anon$1.lookupRelation(HiveContext.scala:149)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$anonfun$apply$2.applyOrElse(Analyzer.scala:83)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$anonfun$apply$2.applyOrElse(Analyzer.scala:81)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:165)
        at org.apache.spark.sql.catalyst.trees.TreeNode$anonfun$4.apply(TreeNode.scala:183)
        at scala.collection.Iterator$anon$11.next(Iterator.scala:328)
        at scala.collection.Iterator$class.foreach(Iterator.scala:727)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
        at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
        at scala.collection.AbstractIterator.to(Iterator.scala:1157)
        at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
        at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
        at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
        at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:212)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:168)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:156)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:81)
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:80)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$anonfun$apply$1$anonfun$apply$2.apply(RuleExecutor.scala:62)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$anonfun$apply$1$anonfun$apply$2.apply(RuleExecutor.scala:60)
        at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
        at scala.collection.immutable.List.foldLeft(List.scala:84)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$anonfun$apply$1.apply(RuleExecutor.scala:60)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor$anonfun$apply$1.apply(RuleExecutor.scala:52)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.apply(RuleExecutor.scala:52)
        at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:313)
        at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:313)
        at org.apache.spark.sql.hive.HiveContext$QueryExecution.optimizedPlan$lzycompute(HiveContext.scala:248)
        at org.apache.spark.sql.hive.HiveContext$QueryExecution.optimizedPlan(HiveContext.scala:247)
        at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan$lzycompute(SQLContext.scala:316)
        at org.apache.spark.sql.SQLContext$QueryExecution.sparkPlan(SQLContext.scala:316)
        at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan$lzycompute(SQLContext.scala:319)
        at org.apache.spark.sql.SQLContext$QueryExecution.executedPlan(SQLContext.scala:319)
        at org.apache.spark.sql.SchemaRDD.collect(SchemaRDD.scala:405)
        at $iwC$iwC$iwC$iwC.<init>(<console>:15)
        at $iwC$iwC$iwC.<init>(<console>:20)
        at $iwC$iwC.<init>(<console>:22)
        at $iwC.<init>(<console>:24)
        at <init>(<console>:26)
        at .<init>(<console>:30)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
        at org.apache.spark.repl.Spark