在创建项目之前确保自己本地安装好了scala环境和java环境,因为spark是scala编写的,scala和java一样都是需要编译成字节码,然后在JVM里面运行。我本地的scala版本是2.11.0版本,hadoop是2.7.6版本
第一步:打开idea,然后创建一个maven项目
在pom里面加入如下依赖:
<properties>
<spark.version>2.3.1</spark.version>
<scala.version>2.11</scala.version>
<geotools.version>20-SNAPSHOT</geotools.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<depend