SparkCore（8）：Spark Standalone和OnYarn两种模式提交命令实例

最新推荐文章于 2022-12-28 22:34:40 发布
最新推荐文章于 2022-12-28 22:34:40 发布
阅读量516 收藏
点赞数
分类专栏： 大数据开发 Spark SparkCore 文章标签： SparkCore SparkOnYarn Standalone
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u010886217/article/details/83317813
版权
大数据开发 同时被 3 个专栏收录
204 篇文章 8 订阅
订阅专栏
Spark
68 篇文章 0 订阅
订阅专栏
SparkCore
18 篇文章 0 订阅
订阅专栏

一、实现功能

二、前提

三、standalone模式

1.前提

2.命令

2.1 client模式

2.2 cluster：提交端口是REST server

四、yarn模式

1.前提

2.命令

2.1 client模式

2.2 cluster模式

一、实现功能

通过将spark的Standalone和OnYarn两种模式集中对比，更容易区分两者区别。

二、前提

1.Standalone和OnYarn两种模式的具体调度流程，可以参考博客

https://blog.csdn.net/u010886217/article/details/101377596

2.本文使用代码是源自：https://blog.csdn.net/u010886217/article/details/83317722，中代码打包至服务器

 

三、standalone模式

1.前提

（1）启动hdfs

（2）启动spark服务（核心！）

sbin/start-all.sh

（3）如果没有指定，则需要查看log，获得提交任务在哪个端口（因为是随机的）：

[root@hadoop logs]# cat /opt/modules/cdh5.7.0/spark-2.1.0-bin-2.6.0-cdh5.7.0/logs/spark-root-org.apache.spark.deploy.master.Master-1-hadoop.out
Spark Command: /opt/jdk1.8.0_151/bin/java -cp /opt/modules/cdh5.7.0/spark-2.1.0-bin-2.6.0-cdh5.7.0/conf/:/opt/modules/cdh5.7.0/spark-2.1.0-bin-2.6.0-cdh5.7.0/jars/*:/opt/modules/cdh5.7.0/hadoop-2.6.0-cdh5.7.0/etc/hadoop/ -Xmx1g org.apache.spark.deploy.master.Master --host hadoop --port 7077 --webui-port 8080
========================================
19/11/14 12:50:45 INFO Master: Started daemon with process name: 4774@hadoop
19/11/14 12:50:45 INFO SignalUtils: Registered signal handler for TERM
19/11/14 12:50:45 INFO SignalUtils: Registered signal handler for HUP
19/11/14 12:50:45 INFO SignalUtils: Registered signal handler for INT
19/11/14 12:50:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/14 12:50:47 INFO SecurityManager: Changing view acls to: root
19/11/14 12:50:47 INFO SecurityManager: Changing modify acls to: root
19/11/14 12:50:47 INFO SecurityManager: Changing view acls groups to: 
19/11/14 12:50:47 INFO SecurityManager: Changing modify acls groups to: 
19/11/14 12:50:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/11/14 12:50:48 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
19/11/14 12:50:48 INFO Master: Starting Spark master at spark://hadoop:7077
19/11/14 12:50:48 INFO Master: Running Spark version 2.1.0
19/11/14 12:50:49 WARN Utils: Service 'MasterUI' could not bind on port 8080. Attempting port 8081.
19/11/14 12:50:49 INFO Utils: Successfully started service 'MasterUI' on port 8081.
19/11/14 12:50:49 INFO MasterWebUI: Bound MasterWebUI to hadoop, and started at http://192.168.0.8:8081
19/11/14 12:50:49 INFO Utils: Successfully started service on port 6066.
19/11/14 12:50:49 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
19/11/14 12:50:50 INFO Master: I have been elected leader! New state: ALIVE
19/11/14 12:50:54 INFO Master: Registering worker 192.168.0.8:58064 with 1 cores, 1388.0 MB RAM

查看 'MasterUI' on port 8081.访问http://192.168.0.8:8081/获得Spark master的提交任务端口7077：

2.命令

2.1 client模式

date=`date +"%Y%m%d%H%M"`
/opt/modules/spark-2.1.0-bin-2.7.3/bin/spark-submit \
--master spark://bigdata.ibeifeng.com:7070 \
--deploy-mode client \
--class _0722rdd.Wordcount_product \
--conf spark.app.coalesce=1 \
/opt/datas/lib/scalaProjectMaven.jar

-》基于wc-hadoop01环境

date=`date +"%Y%m%d%H%M"`
/opt/modules/cdh5.7.0/spark-2.1.0-bin-2.6.0-cdh5.7.0/bin/spark-submit \
--master spark://hadoop:7077 \
--deploy-mode client \
--class _0722rdd.Wordcount_product \
--conf spark.app.coalesce=1 \
/opt/project/scalaproject/scalaProjectMaven.jar

wc-hadoop01运行结果：

19/11/14 18:18:08 INFO SparkContext: Running Spark version 2.1.0
19/11/14 18:18:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/14 18:18:09 INFO SecurityManager: Changing view acls to: root
19/11/14 18:18:09 INFO SecurityManager: Changing modify acls to: root
19/11/14 18:18:09 INFO SecurityManager: Changing view acls groups to: 
19/11/14 18:18:09 INFO SecurityManager: Changing modify acls groups to: 
19/11/14 18:18:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
19/11/14 18:18:09 INFO Utils: Successfully started service 'sparkDriver' on port 41408.
19/11/14 18:18:09 INFO SparkEnv: Registering MapOutputTracker
19/11/14 18:18:09 INFO SparkEnv: Registering BlockManagerMaster
19/11/14 18:18:09 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/11/14 18:18:09 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/11/14 18:18:10 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-b70893fc-caa4-4500-9e99-11aaa41e8cfe
19/11/14 18:18:10 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
19/11/14 18:18:10 INFO SparkEnv: Registering OutputCommitCoordinator
19/11/14 18:18:10 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
19/11/14 18:18:10 INFO Utils: Successfully started service 'SparkUI' on port 4041.
19/11/14 18:18:10 INFO SparkUI: Bound SparkUI to hadoop, and started at http://192.168.0.8:4041
19/11/14 18:18:10 INFO SparkContext: Added JAR file:/opt/project/scalaproject/scalaProjectMaven.jar at spark://192.168.0.8:41408/jars/scalaProjectMaven.jar with timestamp 1573726690750
19/11/14 18:18:10 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://hadoop:7077...
19/11/14 18:18:11 INFO TransportClientFactory: Successfully created connection to hadoop/192.168.0.8:7077 after 68 ms (0 ms spent in bootstraps)
19/11/14 18:18:11 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20191114181811-0000
19/11/14 18:18:11 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40505.
19/11/14 18:18:11 INFO NettyBlockTransferService: Server created on 192.168.0.8:40505
19/11/14 18:18:11 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/11/14 18:18:11 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.0.8, 40505, None)
19/11/14 18:18:11 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.8:40505 with 413.9 MB RAM, BlockManagerId(driver, 192.168.0.8, 40505, None)
19/11/14 18:18:11 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.0.8, 40505, None)
19/11/14 18:18:11 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.0.8, 40505, None)
19/11/14 18:18:11 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20191114181811-0000/0 on worker-20191114125053-192.168.0.8-58064 (192.168.0.8:58064) with 1 cores
19/11/14 18:18:11 INFO StandaloneSchedulerBackend: Granted executor ID app-20191114181811-0000/0 on hostPort 192.168.0.8:58064 with 1 cores, 1024.0 MB RAM
19/11/14 18:18:12 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20191114181811-0000/0 is now RUNNING
19/11/14 18:18:12 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
19/11/14 18:18:14 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 225.8 KB, free 413.7 MB)
19/11/14 18:18:14 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 21.3 KB, free 413.7 MB)
19/11/14 18:18:14 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.0.8:40505 (size: 21.3 KB, free: 413.9 MB)
19/11/14 18:18:14 INFO SparkContext: Created broadcast 0 from textFile at Wordcount_product.scala:34
19/11/14 18:18:16 INFO FileInputFormat: Total input paths to process : 1
19/11/14 18:18:16 INFO SparkContext: Starting job: sortByKey at Wordcount_product.scala:36
19/11/14 18:18:17 INFO DAGScheduler: Registering RDD 3 (map at Wordcount_product.scala:35)
19/11/14 18:18:17 INFO DAGScheduler: Got job 0 (sortByKey at Wordcount_product.scala:36) with 2 output partitions
19/11/14 18:18:17 INFO DAGScheduler: Final stage: ResultStage 1 (sortByKey at Wordcount_product.scala:36)
19/11/14 18:18:17 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
19/11/14 18:18:17 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
19/11/14 18:18:17 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at Wordcount_product.scala:35), which has no missing parents
19/11/14 18:18:17 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.8 KB, free 413.7 MB)
19/11/14 18:18:17 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.8 KB, free 413.7 MB)
19/11/14 18:18:17 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.0.8:40505 (size: 2.8 KB, free: 413.9 MB)
19/11/14 18:18:17 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:996
19/11/14 18:18:17 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at Wordcount_product.scala:35)
19/11/14 18:18:17 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
19/11/14 18:18:18 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.0.8:35601) with ID 0
19/11/14 18:18:18 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 192.168.0.8, executor 0, partition 0, PROCESS_LOCAL, 6072 bytes)
19/11/14 18:18:18 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.0.8:59810 with 413.9 MB RAM, BlockManagerId(0, 192.168.0.8, 59810, None)
19/11/14 18:18:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.0.8:59810 (size: 2.8 KB, free: 413.9 MB)
19/11/14 18:18:20 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.0.8:59810 (size: 21.3 KB, free: 413.9 MB)
19/11/14 18:18:22 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 192.168.0.8, executor 0, partition 1, PROCESS_LOCAL, 6072 bytes)
19/11/14 18:18:22 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3699 ms on 192.168.0.8 (executor 0) (1/2)
19/11/14 18:18:22 INFO DAGScheduler: ShuffleMapStage 0 (map at Wordcount_product.scala:35) finished in 4.750 s
19/11/14 18:18:22 INFO DAGScheduler: looking for newly runnable stages
19/11/14 18:18:22 INFO DAGScheduler: running: Set()
19/11/14 18:18:22 INFO DAGScheduler: waiting: Set(ResultStage 1)
19/11/14 18:18:22 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 233 ms on 192.168.0.8 (executor 0) (2/2)
19/11/14 18:18:22 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
19/11/14 18:18:22 INFO DAGScheduler: failed: Set()
19/11/14 18:18:22 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[7] at sortByKey at Wordcount_product.scala:36), which has no missing parents
19/11/14 18:18:22 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.1 KB, free 413.7 MB)
19/11/14 18:18:22 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.4 KB, free 413.7 MB)
19/11/14 18:18:22 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.0.8:40505 (size: 2.4 KB, free: 413.9 MB)
19/11/14 18:18:22 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:996
19/11/14 18:18:22 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 1 (MapPartitionsRDD[7] at sortByKey at Wordcount_product.scala:36)
19/11/14 18:18:22 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
19/11/14 18:18:22 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, 192.168.0.8, executor 0, partition 0, NODE_LOCAL, 5818 bytes)
19/11/14 18:18:22 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.0.8:59810 (size: 2.4 KB, free: 413.9 MB)
19/11/14 18:18:22 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.0.8:35601
19/11/14 18:18:22 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 154 bytes
19/11/14 18:18:22 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, 192.168.0.8, executor 0, partition 1, NODE_LOCAL, 5818 bytes)
19/11/14 18:18:22 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 352 ms on 192.168.0.8 (executor 0) (1/2)
19/11/14 18:18:22 INFO DAGScheduler: ResultStage 1 (sortByKey at Wordcount_product.scala:36) finished in 0.434 s
19/11/14 18:18:22 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 115 ms on 192.168.0.8 (executor 0) (2/2)
19/11/14 18:18:22 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
19/11/14 18:18:22 INFO DAGScheduler: Job 0 finished: sortByKey at Wordcount_product.scala:36, took 6.205353 s
19/11/14 18:18:23 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
19/11/14 18:18:23 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
19/11/14 18:18:23 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
19/11/14 18:18:23 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
19/11/14 18:18:23 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
19/11/14 18:18:23 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
19/11/14 18:18:23 INFO SparkContext: Starting job: saveAsTextFile at Wordcount_product.scala:46
19/11/14 18:18:23 INFO DAGScheduler: Registering RDD 5 (map at Wordcount_product.scala:36)
19/11/14 18:18:23 INFO DAGScheduler: Got job 1 (saveAsTextFile at Wordcount_product.scala:46) with 1 output partitions
19/11/14 18:18:23 INFO DAGScheduler: Final stage: ResultStage 4 (saveAsTextFile at Wordcount_product.scala:46)
19/11/14 18:18:23 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 3)
19/11/14 18:18:23 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 3)
19/11/14 18:18:23 INFO DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[5] at map at Wordcount_product.scala:36), which has no missing parents
19/11/14 18:18:23 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 192.168.0.8:40505 in memory (size: 2.4 KB, free: 413.9 MB)
19/11/14 18:18:23 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 192.168.0.8:59810 in memory (size: 2.4 KB, free: 413.9 MB)
19/11/14 18:18:23 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 4.0 KB, free 413.7 MB)
19/11/14 18:18:23 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.4 KB, free 413.7 MB)
19/11/14 18:18:23 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.0.8:40505 (size: 2.4 KB, free: 413.9 MB)
19/11/14 18:18:23 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:996
19/11/14 18:18:23 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[5] at map at Wordcount_product.scala:36)
19/11/14 18:18:23 INFO TaskSchedulerImpl: Adding task set 3.0 with 2 tasks
19/11/14 18:18:23 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 4, 192.168.0.8, executor 0, partition 0, NODE_LOCAL, 5813 bytes)
19/11/14 18:18:23 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.0.8:59810 (size: 2.4 KB, free: 413.9 MB)
19/11/14 18:18:23 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 5, 192.168.0.8, executor 0, partition 1, NODE_LOCAL, 5813 bytes)
19/11/14 18:18:23 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 4) in 141 ms on 192.168.0.8 (executor 0) (1/2)
19/11/14 18:18:23 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 5) in 130 ms on 192.168.0.8 (executor 0) (2/2)
19/11/14 18:18:23 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool 
19/11/14 18:18:23 INFO DAGScheduler: ShuffleMapStage 3 (map at Wordcount_product.scala:36) finished in 0.255 s
19/11/14 18:18:23 INFO DAGScheduler: looking for newly runnable stages
19/11/14 18:18:23 INFO DAGScheduler: running: Set()
19/11/14 18:18:23 INFO DAGScheduler: waiting: Set(ResultStage 4)
19/11/14 18:18:23 INFO DAGScheduler: failed: Set()
19/11/14 18:18:23 INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[11] at saveAsTextFile at Wordcount_product.scala:46), which has no missing parents
19/11/14 18:18:23 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 69.3 KB, free 413.6 MB)
19/11/14 18:18:23 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 25.4 KB, free 413.6 MB)
19/11/14 18:18:23 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.0.8:40505 (size: 25.4 KB, free: 413.9 MB)
19/11/14 18:18:23 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:996
19/11/14 18:18:23 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[11] at saveAsTextFile at Wordcount_product.scala:46)
19/11/14 18:18:23 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
19/11/14 18:18:23 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 6, 192.168.0.8, executor 0, partition 0, NODE_LOCAL, 6118 bytes)
19/11/14 18:18:23 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.0.8:59810 (size: 25.4 KB, free: 413.9 MB)
19/11/14 18:18:24 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 192.168.0.8:35601
19/11/14 18:18:24 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 151 bytes
19/11/14 18:18:24 INFO DAGScheduler: ResultStage 4 (saveAsTextFile at Wordcount_product.scala:46) finished in 0.985 s
19/11/14 18:18:24 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 6) in 988 ms on 192.168.0.8 (executor 0) (1/1)
19/11/14 18:18:24 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool 
19/11/14 18:18:24 INFO DAGScheduler: Job 1 finished: saveAsTextFile at Wordcount_product.scala:46, took 1.391035 s

2.2 cluster：提交端口是REST server

date=`date +"%Y%m%d%H%M"`
/opt/modules/spark-2.1.0-bin-2.7.3/bin/spark-submit \
--master spark://bigdata.ibeifeng.com:6066 \
--deploy-mode cluster \
--class _0722rdd.Wordcount_product \
--conf spark.app.coalesce=1 \
/opt/datas/lib/scalaProjectMaven.jar

-》基于wc-hadoop01环境，REST server 端口也是6066

date=`date +"%Y%m%d%H%M"`
/opt/modules/cdh5.7.0/spark-2.1.0-bin-2.6.0-cdh5.7.0/bin/spark-submit \
--master spark://hadoop:6066 \
--deploy-mode cluster \
--class _0722rdd.Wordcount_product \
--conf spark.app.coalesce=1 \
/opt/project/scalaproject/scalaProjectMaven.jar

结果：

Running Spark using the REST application submission protocol.
19/11/14 18:34:26 INFO RestSubmissionClient: Submitting a request to launch an application in spark://hadoop:6066.
19/11/14 18:34:27 INFO RestSubmissionClient: Submission successfully created as driver-20191114183427-0001. Polling submission state...
19/11/14 18:34:27 INFO RestSubmissionClient: Submitting a request for the status of submission driver-20191114183427-0001 in spark://hadoop:6066.
19/11/14 18:34:27 INFO RestSubmissionClient: State of driver driver-20191114183427-0001 is now SUBMITTED.
19/11/14 18:34:27 INFO RestSubmissionClient: Server responded with CreateSubmissionResponse:
{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20191114183427-0001",
  "serverSparkVersion" : "2.1.0",
  "submissionId" : "driver-20191114183427-0001",
  "success" : true
}

 

四、yarn模式

参考：http://spark.apache.org/docs/2.1.0/running-on-yarn.html

1.前提

（1）开启hdfs

（2）开启yarn

2.OnYarn的client和cluster区别

（1）Client

Driver运行在Client端（提交Spark作业的机器）

Client会和请求到的Container进行通信来完成作业的调度和执行，Client是不能退出，如果退出任务结束。

运行日志，直接在客户端打印

（2）Cluster

Driver运行在ApplicationMaster中

Client提交作业后就可以关闭，因为已经在Yarn上运行。

运行日志，需要通过yarn命令查看：

参考：http://spark.apache.org/docs/2.1.0/running-on-yarn.html

yarn logs -applicationId <app ID>

3.命令

3.1 client模式

date=`date +"%Y%m%d%H%M"`
/opt/modules/spark-2.1.0-bin-2.7.3/bin/spark-submit \
--master yarn \
--deploy-mode client \
--class _0722rdd.Wordcount_product \
--conf spark.app.coalesce=1 \
/opt/datas/lib/scalaProjectMaven.jar

-》基于wc-hadoop01环境

参考上一篇文章：https://blog.csdn.net/u010886217/article/details/83317722

date=`date +"%Y%m%d%H%M"`
/opt/modules/cdh5.7.0/spark-2.1.0-bin-2.6.0-cdh5.7.0/bin/spark-submit \
--master yarn \
--deploy-mode client \
--class _0722rdd.Wordcount_product \
--driver-memory   1G \
--driver-cores 1 \
--executor-memory 1G \
--executor-cores 1 \
--num-executors 1 \
--conf spark.app.coalesce=1 \
/opt/project/scalaproject/scalaProjectMaven.jar

3.2 cluster模式

date=`date +"%Y%m%d%H%M"`
/opt/modules/spark-2.1.0-bin-2.7.3/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--class _0722rdd.Wordcount_product \
--conf spark.app.coalesce=1 \
/opt/datas/lib/scalaProjectMaven.jar

-》基于wc-hadoop01环境

date=`date +"%Y%m%d%H%M"`
/opt/modules/cdh5.7.0/spark-2.1.0-bin-2.6.0-cdh5.7.0/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--class _0722rdd.Wordcount_product \
--conf spark.app.coalesce=1 \
/opt/project/scalaproject/scalaProjectMaven.jar

wc-hadoop01运行结果

19/11/14 18:39:26 INFO Client: 
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: 192.168.0.8
         ApplicationMaster RPC port: 0
         queue: root.root
         start time: 1573727952427
         final status: UNDEFINED
         tracking URL: http://hadoop:8088/proxy/application_1573707031333_0002/
         user: root

（测试，成功~）

 

 

RayBreslin
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Spark的安装（Standalone模式，高可用模式，基于Yarn模式）
01-20
spark的Standalone模式安装  一、安装流程 1、将spark-2.2.0-bin-hadoop2.7.tgz 上传到 /usr/local/spark/ 下，然后解压 2、进入到conf中修改名字 改为 .sh 结尾的 3、编辑 spark-env.sh 4、修改slaves 的...
spark报错——无法访问WebUI
weixin_63507910的博客
04-17 1298
并且spark启动无报错，shell登入正常，唯独无法访问WebUI，通过日志最后发现，并非无法访问WebUI，而是spark在启动中，无法绑定到端口8080，导致spark绑定到其他端口，所以无法通过8080访问WebUI。同时，无法访问WebUI也有很多原因，如进程启动失败等导致，建议读者通过日志记录查找问题，最后通过搜索引擎解决问题。笔者在安装spark，并配置spark高可用后，发现无法访问spark的WebUI，即无法访问。关于进程启动失败，而无法访问WebUI，请看笔者这篇文章。
Spark环境搭建（Stand alone模式 && Stand alone HA模式）
m0_48639280的博客
12-28 1554
Standalone模式是Spark自带的一种集群模式，不同于前面本地模式启动多个进程来模拟集群的环境，Standalone模式是真实地在多个机器之间搭建Spark集群的环境，完全可以利用该模式搭建多机器集群，用于实际的大数据处理。开启两个任务，就会在master进程中开启两个对应的 Driver，对应的Executor各自汇报给对应的Driver，Executor数量根据你任务情况而定。在 standalone 模式下， master进程和 worker进程是固定的。
BlockManager原理和源码解密
sundujing的博客
05-15 3026
一、BlockManager运行实例观察从Application启动的角度来观察BlockManager；1，在Application启动时会在SparkEnv中注册BlockManagerMaster以及MapOutputTracker，其中a) BlockManagerMaster：对整个集群的Block数据进行管理的；b) MapOutputTracker:跟踪所有mapper的输出的；2，B
Spark StandAlone环境部署
qq_34999090的博客
03-19 825
1、集群规划 使用三台Linux虚拟机来组成集群环境, 非别是: node1\ node2\ node3 node1运行: Spark的Master进程 和 1个Worker进程 node2运行: spark的1个worker进程 node3运行: spark的1个worker进程 在所有机器安装Python(Anaconda):参考：Anaconda On Linux安装_沉默鹰_90的博客-CSDN博客 在所有机器配置环境变量：参考(以下文档的3、4、5点；Spark先不要安装，最后从no
大规模数据处理中拒绝连接错误分析处理
大师兄你家猴跑啦的博客
07-04 4461
1、处理的数据有几百个G,把数据处理成按照手机号计算1万多个特征 ；2、数据处理环境：     spark-2.0.2;    --executor-memory 40g --total-executor-cores 120 --driver-memory 40g  3、报的错误org.apache.spark.shuffle.MetadataFetchFailedException: Missi...
Spark实验：Standalone模式安装部署（带答案）1
08-03
2. Spark Standalone安装配置：理解Spark在Standalone模式下的配置项和作用。 **三、知识点** 1. Linux常用命令：如tar解压、vi/vim编辑、source使配置生效、jps查看Java进程等。 2. `.bash_profile`配置：通过...
flink-spark-submiter:从本地IDEA提交FlinkSpark任务到Yarnk8s集群
04-13
Flink任务、Spark任务提交到集群，通常...支持Flink yarnPerJob、Standalone 、yarnSession模式下的任务提交。 支持Spark任务以Yarn Cluster模式提交到YARN，支持自动上传用户Jar包，依赖的Spark Jars需要提前上传到H
sparkcore相关实例
最新发布
04-08
Spark Core可与多种资源管理器集成，如YARN、Mesos和Standalone。这些集群管理器负责Executor的生命周期管理和资源分配。 以上是Spark Core的一些核心概念和特性，实际使用中还需要结合具体场景进行调优，如调整...
spark初始化源码阅读sparkonyarn的client和cluster区别
10-20
Spark 初始化源码阅读 Spark on YARN 的 Client 和 Cluster 区别 Spark 是一个大数据处理的开源框架，它可以在多种集群管理器上运行，如 YARN、Mesos 和 Standalone。Spark on YARN 是 Spark 在 YARN 集群管理器上...
Spark：遇到问题汇总
不花的花和尚的博客
10-07 539
问题一：Map output statuses were bytes which exceeds spark.akka.frameSize   17/10/12 00:15:38 INFO spark.MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 1 to spark...
Error communicating with MapOutputTracker-问题解析2
liliang199的专栏
10-12 2652
spark.MapOutputTrackerMasterActor: Map output statuses were 14371441 bytes which exceeds spark.akka.frameSize (10485760 bytes). org.apache.spark.SparkException: Map output statuses were 14371441 bytes which exceeds spark.akka.frameSize (10485760 bytes).
Spark开发中遇到的常见问题以及解决方案（一）
zhangzeyuan56的博客
05-18 8667
问题一：ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkDriver-akka.remote.default-remote-dispatcher-8] shutting down ActorSystem [sparkDriver]18/05/18 15:46:59 ERROR ActorSystemImpl: Uncaug...
Spark Shuffle源码分析系列之MapOutputTracker
进击的数据小白
10-22 732
MapOutputTracker是spark环境的主要组件之一，其功能有三方面，首先DAGScheduler使用MapOutputTrackerMaster来管理各个ShuffleMapTask的输出数据[MapStatus]，另外根据各ShuffleMapTask结果的统计信息来进行尽可能的本地化Scheduler；其次ShuffleMapStage使用MapOutputTrackerMaster来判断是否要进行ShuffleMapTask的计算；最后ShuffleReduce任务用来获取从哪些exec.
spark 的reduce操作
跟我一起去征服
05-03 8796
spark的reduce操作，输入是两个值，输出是一个值，第一第二个值的结果跟第三个值相加，然后前n个数的值和第n个值相加 见代码如下：     public static void myReduce(){         SparkConf conf=new SparkConf()         .setMaster("local")         .setAppName("myR
spark打包程序提交任务
知其然，知其所以然
01-22 887
美图欣赏： 一.操作： 拷贝一个全类名： 程序： import org.apache.spark.{SparkConf, SparkContext} /** * * Scala的word count * */ object WordCountScala extends App { //获取spark的环境,setAppName定义应用程序的名字，s...
“Service 'Driver' could not bind on port”错误的解决方法
坐忘峰
09-11 5762
在deploy model为“cluster”时，出现这个错误，按照网上的各种方式都是了，比如以下方式： 1、修改export SPARK_LOCAL_IP=127.0.0.1; 2、保证SPARK_LOCAL_IP为本机真实IP; 3、关闭防火墙和SELINUX 4、关闭IPV6 等等都不行，最后同事在spark-evn.sh中，通过以下设置方式问题解决了。 export SPARK_LOCAL_
spark master web ui 端口8080被占用解决方法
duncan
07-15 925
spark master web ui 默认端口为8080，当系统有其它程序也在使用该接口时，启动master时也不会报错，spark自己会改用其它端口，自动端口号加1，但为了可以控制到指定的端口，我们可以自行设置，修改方法：   1、cd SPARK_HOME/sbin   2、vi start-master.sh     3、定位到下面部分内容：   if [ "$SPA...
spark解决报错WARN util.Utils: Service 'sparkWorker' could not bind on port 0. Attempting port 1.错误的解决方法
qq_41026556的博客
08-29 2747
需要在spark-defaults.conf文件中设置：spark.ui.port=4040，spark-env.sh中设置export SPARK_LOCAL_IP=master(主机名)。就可以了。
Spark实战：深入理解Standalone与YARN模式
首先，Spark的集群模式主要分为两种：Standalone模式和YARN（Hadoop Yet Another Resource Negotiator）模式。在Standalone模式下，Spark自己管理资源调度，构建了一个主从结构的系统，包括Master节点和Worker节点。...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值