spark1.6升级2.3读取hive表报错NoSuchDatabaseException: Database 'ods' not found

最新推荐文章于 2023-01-06 14:43:59 发布

zhiaixiatian

最新推荐文章于 2023-01-06 14:43:59 发布

阅读量2.2k

点赞数 1

分类专栏： spark 文章标签： sparkstreaming hive

本文链接：https://blog.csdn.net/zhiaixiatian/article/details/102824999

版权

spark 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

升级完spark后，通过spark-submit 操作hive报错，如下：

org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'ods' not found;
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.org$apache$spark$sql$catalyst$catalog$SessionCatalog$$requireDbExists(SessionCatalog.scala:174)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.setCurrentDatabase(SessionCatalog.scala:256)
        at org.apache.spark.sql.execution.command.SetDatabaseCommand.run(databases.scala:59)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
        at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
        at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
        at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3259)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
        at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3258)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:190)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:75)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)

搜索后有的说是

SparkSession没有加 enableHiveSupport()，但是我的已经加了

但是spark-shell可以操作hive，通过对比环境，发现spark-shell的spark.sql.catalogImplementation=hive

而spark-submit后的没有这个配置，而此配置默认是im-memory，并不是hive

于是修改submit配置添加--conf spark.sql.catalogImplementation=hive后，问题解决。