3.saprk集群hdfstest

基于spark的测试

对50g数据进行Hdfstest,task,executor划分

 

 

19/12/27 17:27:37 INFO spark.SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1164

19/12/27 17:27:37 INFO scheduler.DAGScheduler: Submitting 410 missing tasks from ResultStage 9 (MapPartitionsRDD[4] at map at HdfsTest.scala:37) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))

19/12/27 17:27:37 INFO cluster.YarnScheduler: Adding task set 9.0 with 410 tasks

 

19/12/27 17:02:33 INFO cluster.YarnScheduler: Removed TaskSet 11.0, whose tasks have all completed, from pool

19/12/27 17:02:33 INFO scheduler.DAGScheduler: ResultStage 11 (sum at HdfsTest.scala:45) finished in 27.827 s

19/12/27 17:02:33 INFO scheduler.DAGScheduler: Job 11 finished: sum at HdfsTest.scala:45, took 27.851985 s

Returned length(s) of: 1373955.0

19/12/27 17:02:33 INFO server.AbstractConnector: Stopped Spark@7072bc39{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

19/12/27 17:02:33 INFO ui.SparkUI: Stopped Spark web UI at http://hyt-bigdata01:4040

19/12/27 17:02:33 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread

19/12/27 17:02:33 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors

19/12/27 17:02:33 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down

19/12/27 17:02:33 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices

(serviceOption=None,

services=List(),

started=false)

19/12/27 17:02:33 INFO cluster.YarnClientSchedulerBackend: Stopped

19/12/27 17:02:33 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

19/12/27 17:02:33 INFO memory.MemoryStore: MemoryStore cleared

19/12/27 17:02:33 INFO storage.BlockManager: BlockManager stopped

19/12/27 17:02:33 INFO storage.BlockManagerMaster: BlockManagerMaster stopped

19/12/27 17:02:33 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

19/12/27 17:02:33 INFO spark.SparkContext: Successfully stopped SparkContext

19/12/27 17:02:33 INFO util.ShutdownHookManager: Shutdown hook called

19/12/27 17:02:33 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-829dbd76-4b21-4e8b-b7e6-77c6ec66223e

19/12/27 17:02:33 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-0e1437db-71e5-4915-b5c1-a1dca5c709df

[root@hyt-bigdata01 jars]#