spark1.6升级2.3读取hive表报错NoSuchDatabaseException: Database 'ods' not found

56 阅读 0 评论 37 点赞

我是靠谱客的博主秀丽哈密瓜，最近开发中收集的这篇文章主要介绍spark1.6升级2.3读取hive表报错NoSuchDatabaseException: Database 'ods' not found，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

升级完spark后，通过spark-submit 操作hive报错，如下：

org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'ods' not found;
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.org$apache$spark$sql$catalyst$catalog$SessionCatalog$$requireDbExists(SessionCatalog.scala:174)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.setCurrentDatabase(SessionCatalog.scala:256)
        at org.apache.spark.sql.execution.command.SetDatabaseCommand.run(databases.scala:59)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
        at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
        at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
        at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
        at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3259)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
        at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3258)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:190)
        at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:75)
        at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)

搜索后有的说是

SparkSession没有加 enableHiveSupport()，但是我的已经加了

但是spark-shell可以操作hive，通过对比环境，发现spark-shell的spark.sql.catalogImplementation=hive

而spark-submit后的没有这个配置，而此配置默认是im-memory，并不是hive

于是修改submit配置添加--conf spark.sql.catalogImplementation=hive后，问题解决。