Spark环境搭建
一、local本地模式
1.下载spark压缩包spark-3.0.1-bin-hadoop2.7.tgz。地址： http://archive.apache.org/dist/spark/
2.解压并重命名
# 解压到指定路径
tar -zxvf spark-3.0.1-bin-hadoop2.7.tgz -C /opt/modules/
# 重命名
mv spark-3.0.1-bin-hadoop2.7 spark-3.0.1
3.测试
二、Standalone-独立集群模式
1.集群规划
hadoop01：master
hadoop02：worker
hadoop03：worker
2.修改配置文件
进入spark-3.0.1/conf目录下
修改slaves文件
# 复制模板文件
cp slaves.template slaves
# 修改，添加
hadoop02
hadoop03
修改spark-env.sh文件
# 复制模板文件
cp spark-env.sh.template spark-env.sh
# 添加
export JAVA_HOME=/opt/modules/jdk1.8
HADOOP_CONF_DIR=/opt/modules/hadoop-2.7.2/etc/hadoop
YARN_CONF_DIR=/opt/modules/hadoop-2.7.2/etc/hadoop
SPARK_MASTER_HOST=hadoop01
SPARK_MASTER_PORT=7077
SPARK_MASTER_WEBUI_PORT=8080
SPARK_WORKER_CORES=1
SPARK_WORKER_MEMORY=1g
3.分发修改后的spark到其他节点
scp -r spark-3.0.1 jiang@hadoop02:$PWD
scp -r spark-3.0.1 jiang@hadoop03:$PWD
4.测试
启动spark-shell统计hdfs上的wordscount.txt文件单词个数
val textFile = sc.textFile("hdfs://192.168.89.10:9000/test/spark/wordscount.txt")
val counts = textFile.flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)
counts.collect
三、Standalone-HA高可用集群模式
1.在二的基础上修改配置文件spark-env.sh
export JAVA_HOME=/opt/modules/jdk1.8
HADOOP_CONF_DIR=/opt/modules/hadoop-2.7.2/etc/hadoop
YARN_CONF_DIR=/opt/modules/hadoop-2.7.2/etc/hadoop
# Spark配置
#SPARK_MASTER_HOST=hadoop01
SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop01:2181,hadoop02:2181,hadoop03:2181 -Dspark.deploy.zookeeper.dir=/spark"
SPARK_MASTER_PORT=7077
SPARK_MASTER_WEBUI_PORT=8080
SPARK_WORKER_CORES=1
SPARK_WORKER_MEMORY=1g
修改完成后分发到各个节点
2.启动hadoop和zookeeper集群
[jiang@hadoop01 modules]$ xcall.sh jps
-------- hadoop01 ----------
4275 NodeManager
4023 DataNode
4378 JobHistoryServer
4621 Jps
3886 NameNode
4575 QuorumPeerMain
-------- hadoop02 ----------
2656 DataNode
2916 NodeManager
2792 ResourceManager
3276 QuorumPeerMain
3325 Jps
-------- hadoop03 ----------
2883 Jps
2839 QuorumPeerMain
2458 DataNode
2570 SecondaryNameNode
2653 NodeManager
[jiang@hadoop01 modules]$
启动spark集群sbin/start-all.sh
[jiang@hadoop01 sbin]$ xcall.sh jps
-------- hadoop01 ----------
4657 Master
4275 NodeManager
4023 DataNode
4378 JobHistoryServer
4730 Jps
3886 NameNode
4575 QuorumPeerMain
-------- hadoop02 ----------
2656 DataNode
3424 Jps
2916 NodeManager
2792 ResourceManager
3276 QuorumPeerMain
3372 Worker
-------- hadoop03 ----------
2934 Worker
2839 QuorumPeerMain
2984 Jps
2458 DataNode
2570 SecondaryNameNode
2653 NodeManager
在hadoop02单独启动master
[jiang@hadoop02 spark-3.0.1]$ sbin/start-master.sh
3.查看进程
主master
副master
模拟主master宕机
kill -9 主master id
4.查看web页面
主master
副master
四、Spark on Yarn
1.配置Hadoop中yarn-site.xml文件
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 指定YARN的ResourceManager的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop02</value>
</property>
<!--设置yarn集群内存分配方案 内存分配方案我没有配置，配置了resourcemanager启动不了 -->
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>20480</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>20480</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ration</name>
<value>2.1</value>
</property>
<!-- 开启日志聚集功能 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
<description>
开启日志聚集功能，任务执行完之后，将日志文件自动上传到文件系统（如HDFS文件系统），
否则通过namenode1:8088页面查看日志文件的时候，会报错
"Aggregation is not enabled. Try the nodemanager at namenode1:54951"
</description>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>302400</value>
<description>
日志文件保存在文件系统（如HDFS文件系统）的最长时间，默认值是-1，即永久有效。 这里配置的值
是：7天 = 3600 * 24 * 7 = 302400
</description>
</property>
<!-- 设置yarn历史服务器地址 -->
<property>
<name>yarn.log.server.url</name>
<value>http://hadoop01:19888/jobhistory/logs</value>
</property>
<!--关掉虚拟内存检查 -->
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
</configuration>
2.配置spark历史服务器和Yarn整合
修改spark conf目录下配置文件spark-defaults.conf
cp spark-defaults.conf.template spark-defaults.conf
vim spark-defaults.conf
# 添加一下内容
spark.eventLog.enabled true
spark.eventLog.dir hdfs://hadoop01:9000/sparklog
spark.eventLog.compress true
spark.yarn.historyServer.address hadoop01:18080
修改spark conf目录下配置文件spark-env.sh
# 添加一下内容
SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://hadoop01:9000/sparklog/ --Dspark.history.fs.cleaner.enabled=true"
手动创建sparklog文件
hdfs dfs -mkdir -p /sparklog
修改spark日志配置文件，修改日志级别
[jiang@hadoop01 conf]$ cp log4j.properties.template log4j.properties
[jiang@hadoop01 conf]$ vim log4j.properties
3.分发文件
spark-env.sh
spark-defaults.conf
log4j.properties
4.配置spark依赖的jar包
在hdfs上创建存储spark依赖的jar包目录
hdfs dfs -mkdir -p /spark/jars/
上传$SPARK_HOME/jars所有jar包到hdfs
hdfs dfs -put /opt/modules/spark-3.0.1/jars/* /spark/jars/
修改spark-defaults.conf文件，添加一下内容
spark.yarn.jars hdfs://hadoop01:9000/spark/jars/*
分发spark-defaults.conf到各个节点
5.启动服务
启动hdfs和yarn服务
start-dfs.sh
start-yarn.sh
或
start-all.sh
启动spark HistoryServer服务
start-history-server.sh
五、Spark on Yarn两种模式
测试
client模式
SPARK_HOME=/opt/modules/spark-3.0.1
${SPARK_HOME}/bin/spark-submit \
--master yarn \
--deploy-mode client \
--driver-memory 512m \
--driver-cores 1 \
--executor-memory 512m \
--num-executors 2 \
--executor-cores 1 \
--class org.apache.spark.examples.SparkPi \
${SPARK_HOME}/examples/jars/spark-examples_2.12-3.0.1.jar \
10
cluster模式
SPARK_HOME=/opt/modules/spark-3.0.1
${SPARK_HOME}/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--driver-memory 512m \
--driver-cores 1 \
--executor-memory 512m \
--num-executors 2 \
--executor-cores 1 \
--class org.apache.spark.examples.SparkPi \
${SPARK_HOME}/examples/jars/spark-examples_2.12-3.0.1.jar \
10
查看web界面 http://192.168.89.11:8088/cluster
六、Spark on Yarn （spark-submit 提交任务）
maven项目pom完整文件
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.jiang</groupId>
<artifactId>spark</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<encoding>UTF-8</encoding>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.12.10</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>3.0.1</version>
</dependency>
</dependencies>
<build>
<!--资源文件夹-->
<sourceDirectory>src/main/scala</sourceDirectory>
<plugins>
<!--声明并引入构建的插件-->
<!--用于编译Scala代码到class-->
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<!-- -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
代码：
package com.jiang.hello
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}
object WordCount {
def main(args: Array[String]): Unit = {
if(args.length < 2){
println("请指定input和output")
System.exit(1)
}
val conf:SparkConf = new SparkConf().setAppName("wc")//.setMaster("local[*]")
val sc:SparkContext = new SparkContext(conf)
sc.setLogLevel("WARN")
//val count = sc.textFile("data/input/wc.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_)
val lines:RDD[String] = sc.textFile(args(0))
val words:RDD[String] = lines.flatMap(_.split(" "))
val wordandons:RDD[(String,Int)] = words.map((_,1))
val count:RDD[(String,Int)] = wordandons.reduceByKey(_+_)
// count.foreach(println)
// count.collect()
// System.setProperties("HADOOP_USER_NAME","root")
count.repartition(1).saveAsTextFile(args(1))
sc.stop()
}
}
打包步骤
依次双击
上传打的jar包到服务器
选中的是我重命名后的jar，上传这个就行。打包会生成两个jar包，一个大的一个小的，要小的那个。
执行一下命令
SPARK_HOME=/opt/modules/spark-3.0.1
${SPARK_HOME}/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--driver-memory 512m \
--driver-cores 1 \
--executor-memory 512m \
--num-executors 2 \
--executor-cores 1 \
--class com.jiang.hello.WordCount \ #在idea右击你的项目类Copy Reference的路径
/opt/modules/testdata/spark/wordcount.jar \ # 你上传jar的路径
hdfs://192.168.89.10:9000/test/spark/wordscount.txt \ # 要统计的在hdfs上的文件路径
hdfs://192.168.89.10:9000/test/spark/output02 # 计算结果存放hdfs的路径
运行完成后会在hdfs://192.168.89.10:9000/test/spark/output02目录下看到结果
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
