cdh6.3.2中2种spark命令行读取hive表数据方法

52fighting

已于 2023-04-30 14:16:07 修改

阅读量819

点赞数

分类专栏： spark 文章标签： spark hive 大数据

于 2022-08-09 15:02:10 首次发布

本文链接：https://blog.csdn.net/qq_25073261/article/details/126247912

版权

spark 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

前提

将hive配置文件hive-site.xml放到spark的conf目录下

cp /etc/hive/conf/hive-site.xml /etc/spark/conf

在页面重启spark

1.sparksession

spark-shell

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://agent01:4040
Spark context available as 'sc' (master = yarn, app id = application_1659692413492_0026).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0-cdh6.3.2
      /_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.sql("show databases").show();
+------------+
|databaseName|
+------------+
|     default|
|  kylin_hive|
|      testdb|
+------------+


scala>

scala> spark.sql("use testdb").show();
++
||
++
++


scala> spark.sql("show tables").show();
+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
|  testdb|     test|      false|
|  testdb|    test2|      false|
+--------+---------+-----------+


scala> spark.sql("select * from testdb.test").show();
+---+----+
| id|name|
+---+----+
|  1|张三|
+---+----+


scala>

2.hivecontext

spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/08/09 14:39:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
Spark context Web UI available at http://agent02:4041
Spark context available as 'sc' (master = yarn, app id = application_1659692413492_0027).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0-cdh6.3.2
      /_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_191)
Type in expressions to have them evaluated.
Type :help for more information.

scala>
scala> import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.HiveContext

scala> val hiveContext = new HiveContext(sc)
warning: there was one deprecation warning; re-run with -deprecation for details
hiveContext: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@6aa27145

scala> hiveContext.sql("show databases").show()
+------------+
|databaseName|
+------------+
|     default|
|  kylin_hive|
|      testdb|
+------------+

scala> hiveContext.sql("use testdb").show()
++
||
++
++

scala> hiveContext.sql("show tables").show()
+--------+---------+-----------+
|database|tableName|isTemporary|
+--------+---------+-----------+
|  testdb|     test|      false|
|  testdb|    test2|      false|
+--------+---------+-----------+

scala> hiveContext.sql("select * from testdb.test").show()
+---+----+
| id|name|
+---+----+
|  1|张三|
+---+----+

scala>