Spark-集群搭建

最新推荐文章于 2022-07-28 17:35:26 发布

oJueQiang123456

最新推荐文章于 2022-07-28 17:35:26 发布

阅读量331

点赞数

分类专栏： Spark

本文链接：https://blog.csdn.net/oJueQiang123456/article/details/87901194

版权

Spark 专栏收录该内容

13 篇文章 0 订阅

订阅专栏

一、三台服务器

172.16.233.137 (主节点，打开8888和7077端口)
172.16.233.138
172.16.233.139

二、搭建(三台机器配置一样)：已spark-1.6.0-bin-hadoop2.6为例

上传spark-1.6.0-bin-hadoop2.6到/data/local/，并解压。

[root@bogon local]# ls
jdk1.8.0_201  spark-1.6.0-bin-hadoop2.6
[root@bogon local]#

进入到conf目录，拷贝slaves.template->slaves，设置从节点

# A Spark Worker will be started on each of the machines listed below.
# 从节点ip
172.16.233.138
172.16.233.139

拷贝spark-env.sh.template->spark-env.sh，并添加

# - SPARK_YARN_DIST_ARCHIVES, Comma separated list of archives to be distributed with the job.
export SPARK_MASTER_IP=172.16.233.137	#主节点ip
export SPARK_MASTER_PORT=7077		#默认任务提交的端口号
export SPARK_WORKER_CORES=1		#核数
export SPARK_WORKER_MEMORY=1g		#分配内存
export SPARK_MASTER_WEBUI_PORT=8888	#web页面访问端口

进入到sbin目录修改spark-config.sh文件，添加JAVA_HOME

# Add the PySpark classes to the PYTHONPATH:
export PYTHONPATH="${SPARK_HOME}/python:${PYTHONPATH}"
export PYTHONPATH="${SPARK_HOME}/python/lib/py4j-0.9-src.zip:${PYTHONPATH}"
export JAVA_HOME=/data/local/jdk1.8.0_201

三、测试

进入到主节点(172.16.233.137) 的sbin目录，执行./start-all.sh

[root@bogon sbin]# ./start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /data/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.master.Master-1-bogon.out
root@172.16.233.139's password: root@172.16.233.138's password: 
172.16.233.139: starting org.apache.spark.deploy.worker.Worker, logging to /data/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-bogon.out
root@172.16.233.138's password: 172.16.233.138: Permission denied, please try again.
root@172.16.233.138's password: 172.16.233.138: Permission denied, please try again.
172.16.233.138: starting org.apache.spark.deploy.worker.Worker, logging to /data/local/spark-1.6.0-bin-hadoop2.6/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-bogon.out
[root@bogon sbin]# jps
1806 Master
1871 Jps
[root@bogon sbin]#

访问：172.16.233.137:8888

四、运行自带的dome (在主节点上发布任务)

bogon:bin chenfenli$ ./spark-submit --master spark://172.16.233.137:7077 --class org.apache.spark.examples.SparkPi ../lib/spark-examples-1.6.0-hadoop2.6.0.jar 100
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/02/24 13:43:58 INFO SparkContext: Running Spark version 1.6.0
......

正在运行：

19/02/24 13:46:41 INFO TaskSetManager: Starting task 72.0 in stage 0.0 (TID 72, bogon, partition 72,PROCESS_LOCAL, 2156 bytes)
19/02/24 13:46:41 INFO TaskSetManager: Finished task 71.0 in stage 0.0 (TID 71) in 547 ms on bogon (71/100)
19/02/24 13:46:41 INFO TaskSetManager: Starting task 73.0 in stage 0.0 (TID 73, bogon, partition 73,PROCESS_LOCAL, 2156 bytes)
19/02/24 13:46:41 INFO TaskSetManager: Finished task 72.0 in stage 0.0 (TID 72) in 57 ms on bogon (72/100)
19/02/24 13:46:42 INFO TaskSetManager: Starting task 74.0 in stage 0.0 (TID 74, bogon, partition 74,PROCESS_LOCAL, 2156 bytes)
19/02/24 13:46:42 INFO TaskSetManager: Finished task 73.0 in stage 0.0 (TID 73) in 435 ms on bogon (73/100)
19/02/24 13:46:42 INFO TaskSetManager: Starting task 75.0 in stage 0.0 (TID 75, bogon, partition 75,PROCESS_LOCAL, 2156 bytes)
19/02/24 13:46:42 INFO TaskSetManager: Finished task 74.0 in stage 0.0 (TID 74) in 65 ms on bogon (74/100)
19/02/24 13:46:45 INFO TaskSetManager: Starting task 76.0 in stage 0.0 (TID 76, bogon, partition 76,PROCESS_LOCAL, 2156 bytes)

运行完成：

19/02/24 13:47:01 INFO TaskSetManager: Finished task 75.0 in stage 0.0 (TID 75) in 19260 ms on bogon (100/100)
19/02/24 13:47:01 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:36) finished in 163.366 s
19/02/24 13:47:01 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
19/02/24 13:47:01 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:36, took 164.838521 s
Pi is roughly 3.1409356
19/02/24 13:47:01 INFO SparkUI: Stopped Spark web UI at http://192.168.1.102:4040
19/02/24 13:47:01 INFO SparkDeploySchedulerBackend: Shutting down all executors
19/02/24 13:47:01 INFO SparkDeploySchedulerBackend: Asking each executor to shut down
19/02/24 13:47:01 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/02/24 13:47:01 INFO MemoryStore: MemoryStore cleared
19/02/24 13:47:01 INFO BlockManager: BlockManager stopped
19/02/24 13:47:01 INFO BlockManagerMaster: BlockManagerMaster stopped
19/02/24 13:47:01 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/02/24 13:47:01 INFO SparkContext: Successfully stopped SparkContext
19/02/24 13:47:02 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
19/02/24 13:47:02 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
19/02/24 13:47:02 INFO ShutdownHookManager: Shutdown hook called
19/02/24 13:47:02 INFO ShutdownHookManager: Deleting directory /private/var/folders/03/kcv338w15nq7b_xlt8bb5t_c0000gn/T/spark-52eb2ce0-f7c6-485a-821e-d88fe3ece043/httpd-1b15550b-9e78-4a9e-ad47-dc361c590e2f
19/02/24 13:47:02 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
19/02/24 13:47:02 INFO ShutdownHookManager: Deleting directory /private/var/folders/03/kcv338w15nq7b_xlt8bb5t_c0000gn/T/spark-52eb2ce0-f7c6-485a-821e-d88fe3ece043
bogon:bin chenfenli$