spark-shell连接hive表

首先，必须让你的spark和hive都处于可以正常运行的状态下

然后，把hive-site.xml文件拷贝到spark的conf目录下

cp hive-site.xml /opt/module/spark-yarn/conf/

        这里需要注意：如果以前hive-site.xml文件中，配置过Tez相关信息，注释掉

接着，把Mysql的驱动copy到Spark的jars/目录下

cp mysql-connector-java-5.1.27-bin.jar /opt/module/spark-yarn/jars/

然后启动hive

[atguigu@hadoop102 spark-yarn]$ hive
which: no hbase in (/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/module/jdk1.8.0_212/bin:/opt/module/hadoop-3.1.3/bin:/opt/module/hadoop-3.1.3/sbin:/opt/module/kafka/bin:/opt/module/spark-standalone/bin:/opt/module/spark-standalone/sbin:/opt/module/hive/bin:/home/atguigu/.local/bin:/home/atguigu/bin:/opt/module/jdk1.8.0_212/bin:/opt/module/hadoop-3.1.3/bin:/opt/module/hadoop-3.1.3/sbin:/opt/module/kafka/bin:/opt/module/spark-yarn/bin:/opt/module/spark-yarn/sbin:/opt/module/hive/bin)
Hive Session ID = bcdc30ed-4ef5-483a-99a5-ccd50344c7f4

Logging initialized using configuration in file:/opt/module/hive/conf/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Hive Session ID = b0063111-1ee5-439f-b770-2b7b8ba69acc
hive (default)> select * from student;
OK
student.id	student.name
1	zhangsan
2	lisi
3	lisi
4	lisi
5	wangwu
Time taken: 1.602 seconds, Fetched: 5 row(s)

启动spark-shell

scala> [atguigu@hadoop102 spark-yarn]$ ./bin/spark-shell --master yarn --deploy-mode client
2023-11-02 16:21:27,668 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2023-11-02 16:21:34,001 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
Spark context Web UI available at http://hadoop102:4040
Spark context available as 'sc' (master = yarn, app id = application_1698894615526_0009).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.0
      /_/
         
Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.sql("select * from student").show
2023-11-02 16:21:53,575 WARN conf.HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
2023-11-02 16:21:54,165 WARN conf.HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
2023-11-02 16:21:54,165 WARN conf.HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
2023-11-02 16:21:54,166 WARN conf.HiveConf: HiveConf of name hive.stats.retries.wait does not exist
2023-11-02 16:21:54,845 WARN conf.HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
+---+--------+                                                                  
| id|    name|
+---+--------+
|  1|zhangsan|
|  2|    lisi|
|  3|    lisi|
|  4|    lisi|
|  5|  wangwu|
+---+--------+

成功！