Spark安装及IDEA第一个Spark程序

Spark安装

Linux下安装

解压

tar -zxvf spark-3.1.2-bin-hadoop3.2.tgz -C /opt/software/

改名

mv spark-3.1.2-bin-hadoop3.2/ spark312

修改conf下配置文件

#释放配置文件
cp spark-env.sh.template spark-env.sh

#配置信息
export SPARK_MASTER_HOST=wantest03
export SPARK_MASTER_PORT=7077

配置系统环境变量

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/opt/software/spark312
export PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH

启动测试

#激活系统环境
source /etc/profile

#测试客户端启动
spark-shell

#测试集群启动
start-master.sh
start-slave.sh spark://wantest03:7077

Windows下安装

系统变量配置

Path变量配置

创建IDEA程序

新建一个maven工程，选择scala-simple

删除test文件夹

修改scala版本,与本地一致

添加依赖

<dependencies>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>${scala.version}</version>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.4</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.specs</groupId>
        <artifactId>specs</artifactId>
        <version>1.2.5</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>3.1.2</version>
    </dependency>
    <dependency>
        <groupId>commons-collections</groupId>
        <artifactId>commons-collections</artifactId>
        <version>3.2.2</version>
    </dependency>
</dependencies>

测试