idea中,Spark SQL连接到了一个部署好的Hive,尝试在 包含Hive支持的Spark SQL中CREATE TABLE
我的代码:
package spark.hive
import org.apache.spark.sql.SparkSession
object SparkOnHive extends App {
case class Record(key: Int, value: String)
//val warehouseLocation = new File("/user/hive/warehouse/").getAbsolutePath
val spark = SparkSession
.builder()
.appName("Spark Hive Example")
.master("local[1]")
.getOrCreate()
import spark.implicits._
import spark.sql
sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)").show()
sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO TABLE src").show
sql("SELECT * FROM src").show()
}
报错如下:
Hive support is required to CREATE Hive TABLE (AS SELECT);; 'CreateTable `sr`,org.apach.hadoop.hive.serde2.lazy.LazySimpleSerDe,Ignore
问题分析:在spark中,必须启用hive支持(即.enableHiveSupport())
解决办法:在spark中添加.enableHiveSupport()
修改代码如下:
val spark = SparkSession
.builder()
.appName("Spark Hive Example")
.master("local[1]")
.enableHiveSupport()
.getOrCreate()
运行程序:
成功。。。