需要安装包直接联系:qq:258167139
1.下载安装spark
- spark下载链接: 点击直接跳转到官网在线下载.
2.传入到服务器
2.1解压文件
# 解压到当前文件夹
tar zxvf spark-3.2.0-bin-hadoop3.2.tgz
# 解压到指定的目spark录(首先创建好目录)
tar -zxvf spark-3.2.0-bin-hadoop3.2.tgz /usr/local/spark
2.2给spark-3.2.0-bin-hadoop3.2
文件夹加一个别名
ln -s /usr/local/spark-3.2.0-bin-hadoop3.2 /usr/local/spark
- 成功标志
3.配置环境变量
#java
export JAVA_HOME=/usr/local/jdk1.8.0_281
export PATH=$PATH:$JAVA_HOME/bin
#HADOOP
export HADOOP_HOME=/usr/local/hadoop-3.3.1
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
#python env
export PATH=/usr/local/python3.8/bin/:$PATH
#spark env
export SPARK_HOME=/usr/local/spark-3.2.0-bin-hadoop3.2
#PYSPARK_PYTHON
export PYSPARK_PYTHON=/usr/local/anaconda3/envs/pyspark/bin/python
- 跑一个程序pyspark
(pyspark) [root@6274master bin]# ./pyspark
Python 3.8.12 (default, Oct 12 2021, 13:49:34)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/01/11 08:17:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.2.0
/_/
Using Python version 3.8.12 (default, Oct 12 2021 13:49:34)
Spark context Web UI available at http://6274master:4040
Spark context available as 'sc' (master = local[*], app id = local-1641907063490).
SparkSession available as 'spark'.
>>> sc.parallelize([1,2,3,4,5]).map(lambda x: x + 1).collect()
[2, 3, 4, 5, 6]
>>>
- 成功标识
- 测试spark-shell
(pyspark) [root@6274master bin]# ./spark-shell
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/01/11 09:28:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://6274master:4040
Spark context available as 'sc' (master = local[*], app id = local-1641911313917).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.2.0
/_/
Using Scala version 2.12.15 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_281)
Type in expressions to have them evaluated.
Type :help for more information.
scala> sc.parallelize(Array(1,2,3,4,5)).map(x=> x + 1).collect()
res0: Array[Int] = Array(2, 3, 4, 5, 6)
- 测试成功示例
- spark-submit提交代码运行
(pyspark) [root@6274master bin]# ./spark-submit --master local[*] /usr/local/spark/examples/src/main/python/pi.py 10
4.修改spark配置文件
4.1配置配置文件
进入到spark的配置文件目录中, cd $SPARK_HOME/conf
4.1.1 配置workers文件
# 改名, 去掉后面的.template后缀
mv workers.template workers
# 编辑worker文件
vim workers
# 将里面的localhost删除, 追加
node1
node2
node3
到workers文件内
# 功能: 这个文件就是指示了 当前SparkStandAlone环境下, 有哪些worker
4.1.2配置spark-env.sh文件
# 1. 改名
mv spark-env.sh.template spark-env.sh
# 2. 编辑spark-env.sh, 在底部追加如下内容
## 设置JAVA安装目录
JAVA_HOME=/export/server/jdk
## HADOOP软件配置文件目录,读取HDFS上文件和运行YARN集群
HADOOP_CONF_DIR=/export/server/hadoop/etc/hadoop
YARN_CONF_DIR=/export/server/hadoop/etc/hadoop
## 指定spark老大Master的IP和提交任务的通信端口
# 告知Spark的master运行在哪个机器上
export SPARK_MASTER_HOST=node1
# 告知sparkmaster的通讯端口
export SPARK_MASTER_PORT=7077
# 告知spark master的 webui端口
SPARK_MASTER_WEBUI_PORT=8080
# worker cpu可用核数
SPARK_WORKER_CORES=1
# worker可用内存
SPARK_WORKER_MEMORY=1g
# worker的工作通讯地址
SPARK_WORKER_PORT=7078
# worker的 webui地址
SPARK_WORKER_WEBUI_PORT=8081
## 设置历史服务器
# 配置的意思是 将spark程序运行的历史日志 存到hdfs的/sparklog文件夹中
SPARK_HISTORY_OPTS="-Dspark.history.fs.logDirectory=hdfs://node1:8020/sparklog/ -Dspark.history.fs.cleaner.enabled=true"
- 注意, 上面的配置的路径 要根据你自己机器实际的路径来写
在HDFS上创建程序运行历史记录存放的文件夹:
hadoop fs -mkdir /sparklog
hadoop fs -chmod 777 /sparklog
4.1.3配置spark-defaults.conf文件
# 1. 改名
mv spark-defaults.conf.template spark-defaults.conf
# 2. 修改内容, 追加如下内容
# 开启spark的日期记录功能
spark.eventLog.enabled true
# 设置spark日志记录的路径
spark.eventLog.dir hdfs://node1:8020/sparklog/
# 设置spark日志是否启动压缩
spark.eventLog.compress true
4.1.4配置log4j.properties 文件 [可选配置]
# 1. 改名
mv log4j.properties.template log4j.properties
# 2. 修改内容 参考下图
这个文件的修改不是必须的, 为什么修改为WARN. 因为Spark是个话痨
会疯狂输出日志, 设置级别为WARN 只输出警告和错误日志, 不要输出一堆废话.
5.给其他集群虚拟机分发已经部署好的spark文件
# 把已经配置好的spark分发到其他虚拟机 分发到第二台虚拟机/usr/local目录下
scp -r spark-3.2.0-bin-hadoop3.2 192.168.47.11:/usr/local/
6. 检查
检查每台机器的:
JAVA_HOME
SPARK_HOME
PYSPARK_PYTHON
等等 环境变量是否正常指向正确的目录
- 首先启动启动历史服务器
sbin/start-history-server.sh
7.启动服务
(base) [root@6274master spark]# sbin/start-all.sh
org.apache.spark.deploy.master.Master running as process 10844. Stop it first.
6274master: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-3.2.0-bin-hadoop3.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-6274master.out
6274slave0: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-3.2.0-bin-hadoop3.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-6274slave0.out
6274slave1: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark-3.2.0-bin-hadoop3.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-6274slave1.out
(base) [root@6274master spark]# jps
10992 Worker
9505 DataNode
9778 SecondaryNameNode
10050 ResourceManager
10234 NodeManager
9323 NameNode
10844 Master
11023 Jps
(base) [root@6274master spark]# sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to /usr/local/spark-3.2.0-bin-hadoop3.2/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-6274master.out
(base) [root@6274master spark]# jps
10992 Worker
9505 DataNode
9778 SecondaryNameNode
10050 ResourceManager
11138 Jps
10234 NodeManager
9323 NameNode
10844 Master
11102 HistoryServer
(base) [root@6274master spark]#
- 启动成功后jps检查,worker,Master,历史服务器HisrtoryServer等节点的启动情况
9.查看Master的WEB UI
默认端口master我们设置到了8080
ip:端口
如果端口被占用, 会顺延到8081 …;8082… 8083… 直到申请到端口为止
可以在日志中查看, 具体顺延到哪个端口上:
Service 'MasterUI' could not bind on port 8080. Attempting port 8081.
- 查看spark webUI
10.链接客户端工具测试
- 链接到集群
(base) [root@6274master bin]# ./pyspark --master spark://192.168.47.10:7077
11跑一个计算单词数量的程序
(base) [root@6274master bin]# ./pyspark --master spark://6274master:7077
Python 3.8.12 (default, Oct 12 2021, 13:49:34)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
22/01/12 04:56:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-jaa classes where applicable
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.2.0
/_/
Using Python version 3.8.12 (default, Oct 12 2021 13:49:34)
Spark context Web UI available at http://6274master:4040
Spark context available as 'sc' (master = spark://6274master:7077, app id = app-20220112045607-0002).
SparkSession available as 'spark'.
>>>
>>> sc.textFile("hdfs://6274master:9000/input/test.txt").flatMap(lambda line: line.split(" ")).map(lambda x:(x,1)).reduceByKey(lambda a, b:a + b).collect()
# 结果
[('hadoop', 2), ('java', 2), ('python', 1), ('R', 1), ('C', 1), ('html', 3), ('c', 1), ('javascripts', 2), ('linux', 2)]
>>>