windows下IDEA使用spark连接hive

1、前言

 由于需要在本地环境连接测试集群相关的一些数据，传统方式需要先打jar包，然后再放在集群上使用spark-submit，这样非常麻烦，所以在网上找了下开发环境使用sparksql连接hive的方法，经过许久的折腾，总算弄成功了，特此记录一下。

2、准备

 A、配置文件

把集群上的hadoop etc/hadoop目录下的core-site.xml   hdfs-site.xml  （读取hdfs上的文件必须）
hive的配置文件 hive-site.xml（读取hive必须）拷贝到resources目录下，

 B、hadoop相关配置文件

下载windows下的hadoop winutils插件，https://github.com/steveloughran/winutils
按照自己的版本来，然后配置环境变量
HADOOP_HOME=
HADOOP_USER=

3、pom.xml引入依赖包

  <dependencies>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-common</artifactId>
      <version>2.7.2</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-core -->
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-mapreduce-client-core</artifactId>
      <version>2.7.2</version>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-repl_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-hive_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>

    <dependency>
      <groupId>mysql</groupId>
      <artifactId>mysql-connector-java</artifactId>
      <version>8.0.21</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>${spark.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>${spark.version}</version>
      <scope>compile</scope>
    </dependency>

  </dependencies>

4、测试代码

package org;
import org.apache.spark.sql.*;

public class SparkToCSVJava {

    public static void main(String[] args) throws ParseException {

        SparkSession spark = SparkSession.builder()
                .appName("ParseCSV")
                .master("local[2]")
                .config("spark.executor.cores", "1")
                .config("spark.executor.memory", "1G")
                .config("hive.exec.dynamic.partition", "true")
                .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
                .config("hive.exec.dynamic.partition.mode", "nonstrict")
                .config("spark.sql.sources.partitionOverwriteMode", "dynamic")
                .config("hive.metastore.uris", "thrift://hadoop103:9083")
                .enableHiveSupport()
                .getOrCreate();

        Dataset df=spark.sql("show databases");

        df.show();
    }
}