1、前言
由于需要在本地环境连接测试集群相关的一些数据,传统方式需要先打jar包,然后再放在集群上使用spark-submit,这样非常麻烦,所以在网上找了下开发环境使用sparksql连接hive的方法,经过许久的折腾,总算弄成功了,特此记录一下。
2、准备
A、配置文件
把集群上的hadoop etc/hadoop目录下的core-site.xml hdfs-site.xml (读取hdfs上的文件必须)
hive的配置文件 hive-site.xml(读取hive必须)拷贝到resources目录下,
B、hadoop相关配置文件
下载windows下的hadoop winutils插件,https://github.com/steveloughran/winutils
按照自己的版本来,然后配置环境变量
HADOOP_HOME=
HADOOP_USER=
3、pom.xml引入依赖包
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-core -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-repl_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.21</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
</dependency>
</dependencies>
4、测试代码
package org;
import org.apache.spark.sql.*;
public class SparkToCSVJava {
public static void main(String[] args) throws ParseException {
SparkSession spark = SparkSession.builder()
.appName("ParseCSV")
.master("local[2]")
.config("spark.executor.cores", "1")
.config("spark.executor.memory", "1G")
.config("hive.exec.dynamic.partition", "true")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("hive.exec.dynamic.partition.mode", "nonstrict")
.config("spark.sql.sources.partitionOverwriteMode", "dynamic")
.config("hive.metastore.uris", "thrift://hadoop103:9083")
.enableHiveSupport()
.getOrCreate();
Dataset df=spark.sql("show databases");
df.show();
}
}