Spark-2.4.5搭建

 

3台：

hadoop集群

名称	IP
hadoop01	192.168.204.130
hadoop02	192.168.204.131
hadoop03	192.168.204.132

目录

1：安装【hadoop01执行】下载并解压

2：【hadoop01执行】修改配置文件名称

3：【hadoop01执行】修改配置文件

1：编辑spark-env.sh

2：编辑slaves

4：分发目录

1：分发

2：【hadoop02、hadoop03执行】设置软链接

5：启动Spark

1：jps查看进程

2：通过WEB管理界面查看集群状态 hadoop01:8081

7：执行任务

---

1：安装【hadoop01执行】下载并解压

cd /mwd
wget https://archive.apache.org/dist/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz
mkdir -p /export/server #hadoop02 hadoop03执行
# 解压
tar -zxvf spark-2.4.5-bin-hadoop2.7.tgz -C /export/server/

# 软链接
ln -s /export/server/spark-2.4.5-bin-hadoop2.7 /export/server/spark

2：【hadoop01执行】修改配置文件名称

# 改名
cd /export/server/spark/conf
mv spark-env.sh.template spark-env.sh
mv slaves.template slaves

3：【hadoop01执行】修改配置文件

1：编辑spark-env.sh

vim spark-env.sh

## 设置JAVA安装目录
JAVA_HOME=/usr/lib/jvm/jdk1.8.0_161

## HADOOP软件配置文件目录，读取HDFS上文件和运行YARN集群
HADOOP_CONF_DIR=/data/hadoop/app/etc/hadoop/
YARN_CONF_DIR=/data/hadoop/app/etc/hadoop/

## 指定spark老大Master的IP和提交任务的通信端口
export SPARK_MASTER_HOST=hadoop01
export SPARK_MASTER_PORT=7077

SPARK_MASTER_WEBUI_PORT=8080
SPARK_WORKER_CORES=1
SPARK_WORKER_MEMORY=1g

2：编辑slaves

vim slaves

hadoop01
hadoop02
hadoop03

4：分发目录

1：分发

cd /export/server/
scp -r spark-2.4.5-bin-hadoop2.7/ root@hadoop02:/export/server/
scp -r spark-2.4.5-bin-hadoop2.7/ root@hadoop03:/export/server/

2：【hadoop02、hadoop03执行】设置软链接

ln -s /export/server/spark-2.4.5-bin-hadoop2.7 /export/server/spark

5：启动Spark

/export/server/spark/sbin/start-all.sh

# 如需停止，可以
/export/server/spark/sbin/stop-all.sh

1：jps查看进程

[root@hadoop01 server]# jps
77683 RunJar
77106 RunJar
1032 Master
80315 NodeManager
1306 Jps
79533 DataNode
79292 NameNode
1149 Worker
80111 ResourceManager

[root@hadoop02 server]# jps
126418 Worker
82885 SecondaryNameNode
83125 NodeManager
82631 DataNode
126775 Jps

[root@hadoop03 server]# jps
73137 NodeManager
72704 DataNode
127079 Jps
126649 Worker

2：通过WEB管理界面查看集群状态 hadoop01:8081

 

7：执行任务

/export/server/spark/bin/spark-submit --master spark://hadoop01:7077 --class org.apache.spark.examples.SparkPi /export/server/spark/examples/jars/spark-examples_2.11-2.4.5.jar