代码如下:
object demo2 {
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf()
//sparkConf.setAppName("HiveContext").setMaster("local[*]")
val sc = new SparkContext(sparkConf)
val hiveContext = new HiveContext(sc)
hiveContext.table("emp").show()
sc.stop()
}
}
hive emp 表如下:
打包代码发送到服务器后,在 spark/bin 目录下执行如下命令:
[hadoop@hadoop2 bin]$ ./spark-submit \
--class sparkSql.demo2 \
--master spark://hadoop102:7077 \
--jars /opt/module/hive-1.2.2/lib/mysql-connector-java-5.1.27-bin.jar \
/home/hadoop/SparkLearning-0.0.1-SNAPSHOT-jar-with-dependencies.jar
运行结果如下,已经读取到 Hive 表 emp 中的数据:
执行过程中报错,将看下面博文排查:
SparkSQL 通过 HiveContext 查询 hive表数据报错 java.lang.ClassNotFoundException: sparkSql.demo2
Spark SQL 与 Hive 交互报错:Specified key was too long; max key length is 3072 bytes
HiveContext 执行报错Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
HiveContext 执行报错Exception in org.apache.spark.sql.AnalysisException: Table or view not found