Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
18/06/20 11:36:08 INFO SparkContext: Running Spark version 2.3.1
18/06/20 11:36:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/06/20 11:36:08 INFO SparkContext: Submitted application: Simple Application
18/06/20 11:36:08 INFO SecurityManager: Changing view acls to: yutao
18/06/20 11:36:08 INFO SecurityManager: Changing modify acls to: yutao
18/06/20 11:36:08 INFO SecurityManager: Changing view acls groups to:
18/06/20 11:36:08 INFO SecurityManager: Changing modify acls groups to:
18/06/20 11:36:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yutao); groups with view permissions: Set(); users with modify permissions: Set(yutao); groups with modify permissions: Set()
18/06/20 11:36:09 INFO Utils: Successfully started service 'sparkDriver' on port 49461.
18/06/20 11:36:09 INFO SparkEnv: Registering MapOutputTracker
18/06/20 11:36:09 INFO SparkEnv: Registering BlockManagerMaster
18/06/20 11:36:09 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/06/20 11:36:09 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/06/20 11:36:09 INFO DiskBlockManager: Created local directory at C:\Users\yutao\AppData\Local\Temp\blockmgr-60f73a77-407c-4d1c-bfe7-56ab00ae390c
18/06/20 11:36:09 INFO MemoryStore: MemoryStore started with capacity 901.8 MB
18/06/20 11:36:09 INFO SparkEnv: Registering OutputCommitCoordinator
18/06/20 11:36:09 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/06/20 11:36:09 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://yutao.go-goal.com:4040
18/06/20 11:36:09 INFO Executor: Starting executor ID driver on host localhost
18/06/20 11:36:09 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 49470.
18/06/20 11:36:09 INFO NettyBlockTransferService: Server created on yutao.go-goal.com:49470
18/06/20 11:36:09 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/06/20 11:36:09 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, yutao.go-goal.com, 49470, None)
18/06/20 11:36:09 INFO BlockManagerMasterEndpoint: Registering block manager yutao.go-goal.com:49470 with 901.8 MB RAM, BlockManagerId(driver, yutao.go-goal.com, 49470, None)
18/06/20 11:36:09 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, yutao.go-goal.com, 49470, None)
18/06/20 11:36:09 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, yutao.go-goal.com, 49470, None)
18/06/20 11:36:09 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/D:/sts/workspace/simple-project/spark-warehouse/').
18/06/20 11:36:09 INFO SharedState: Warehouse path is 'file:/D:/sts/workspace/simple-project/spark-warehouse/'.
18/06/20 11:36:10 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
18/06/20 11:36:13 INFO FileSourceStrategy: Pruning directories with:
18/06/20 11:36:14 INFO FileSourceStrategy: Post-Scan Filters:
18/06/20 11:36:14 INFO FileSourceStrategy: Output Data Schema: struct
18/06/20 11:36:14 INFO FileSourceScanExec: Pushed Filters:
18/06/20 11:36:15 INFO CodeGenerator: Code generated in 353.178554 ms
18/06/20 11:36:15 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 220.1 KB, free 901.6 MB)
18/06/20 11:36:16 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 20.6 KB, free 901.6 MB)
18/06/20 11:36:16 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on yutao.go-goal.com:49470 (size: 20.6 KB, free: 901.8 MB)
18/06/20 11:36:16 INFO SparkContext: Created broadcast 0 from cache at SimpleApp.java:14
18/06/20 11:36:16 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4198113 bytes, open cost is considered as scanning 4194304 bytes.
18/06/20 11:36:19 INFO CodeGenerator: Code generated in 37.243378 ms
18/06/20 11:36:19 INFO CodeGenerator: Code generated in 39.624506 ms
18/06/20 11:36:19 INFO SparkContext: Starting job: count at SimpleApp.java:15
18/06/20 11:36:20 INFO DAGScheduler: Registering RDD 7 (count at SimpleApp.java:15)
18/06/20 11:36:20 INFO DAGScheduler: Got job 0 (count at SimpleApp.java:15) with 1 output partitions
18/06/20 11:36:20 INFO DAGScheduler: Final stage: ResultStage 1 (count at SimpleApp.java:15)
18/06/20 11:36:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
18/06/20 11:36:20 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
18/06/20 11:36:20 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[7] at count at SimpleApp.java:15), which has no missing parents
18/06/20 11:36:20 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 18.2 KB, free 901.5 MB)
18/06/20 11:36:20 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 8.4 KB, free 901.5 MB)
18/06/20 11:36:20 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on yutao.go-goal.com:49470 (size: 8.4 KB, free: 901.8 MB)
18/06/20 11:36:20 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1039
18/06/20 11:36:20 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[7] at count at SimpleApp.java:15) (first 15 tasks are for partitions Vector(0))
18/06/20 11:36:20 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
18/06/20 11:36:20 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 8316 bytes)
18/06/20 11:36:20 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
18/06/20 11:36:21 INFO FileScanRDD: Reading File path: file:///D:/Program%20Files/spark-2.3.1-bin-hadoop2.7/README.md, range: 0-3809, partition values: [empty row]
18/06/20 11:36:21 INFO CodeGenerator: Code generated in 22.258382 ms
18/06/20 11:36:21 INFO MemoryStore: Block rdd_2_0 stored as values in memory (estimated size 4.4 KB, free 901.5 MB)
18/06/20 11:36:21 INFO BlockManagerInfo: Added rdd_2_0 in memory on yutao.go-goal.com:49470 (size: 4.4 KB, free: 901.8 MB)
18/06/20 11:36:21 INFO CodeGenerator: Code generated in 5.141693 ms
18/06/20 11:36:21 INFO CodeGenerator: Code generated in 62.976727 ms
18/06/20 11:36:22 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1984 bytes result sent to driver
18/06/20 11:36:22 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1665 ms on localhost (executor driver) (1/1)
18/06/20 11:36:22 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
18/06/20 11:36:22 INFO DAGScheduler: ShuffleMapStage 0 (count at SimpleApp.java:15) finished in 2.036 s
18/06/20 11:36:22 INFO DAGScheduler: looking for newly runnable stages
18/06/20 11:36:22 INFO DAGScheduler: running: Set()
18/06/20 11:36:22 INFO DAGScheduler: waiting: Set(ResultStage 1)
18/06/20 11:36:22 INFO DAGScheduler: failed: Set()
18/06/20 11:36:22 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[10] at count at SimpleApp.java:15), which has no missing parents
18/06/20 11:36:22 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 7.4 KB, free 901.5 MB)
18/06/20 11:36:22 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 3.8 KB, free 901.5 MB)
18/06/20 11:36:22 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on yutao.go-goal.com:49470 (size: 3.8 KB, free: 901.8 MB)
18/06/20 11:36:22 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1039
18/06/20 11:36:22 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[10] at count at SimpleApp.java:15) (first 15 tasks are for partitions Vector(0))
18/06/20 11:36:22 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
18/06/20 11:36:22 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, ANY, 7754 bytes)
18/06/20 11:36:22 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
18/06/20 11:36:22 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
18/06/20 11:36:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 26 ms
18/06/20 11:36:22 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1782 bytes result sent to driver
18/06/20 11:36:22 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 294 ms on localhost (executor driver) (1/1)
18/06/20 11:36:22 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
18/06/20 11:36:22 INFO DAGScheduler: ResultStage 1 (count at SimpleApp.java:15) finished in 0.304 s
18/06/20 11:36:22 INFO DAGScheduler: Job 0 finished: count at SimpleApp.java:15, took 3.013115 s
18/06/20 11:36:23 INFO SparkContext: Starting job: count at SimpleApp.java:16
18/06/20 11:36:23 INFO DAGScheduler: Registering RDD 15 (count at SimpleApp.java:16)
18/06/20 11:36:23 INFO DAGScheduler: Got job 1 (count at SimpleApp.java:16) with 1 output partitions
18/06/20 11:36:23 INFO DAGScheduler: Final stage: ResultStage 3 (count at SimpleApp.java:16)
18/06/20 11:36:23 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 2)
18/06/20 11:36:23 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 2)
18/06/20 11:36:23 INFO DAGScheduler: Submitting ShuffleMapStage 2 (MapPartitionsRDD[15] at count at SimpleApp.java:16), which has no missing parents
18/06/20 11:36:23 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 18.2 KB, free 901.5 MB)
18/06/20 11:36:23 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 8.4 KB, free 901.5 MB)
18/06/20 11:36:23 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on yutao.go-goal.com:49470 (size: 8.4 KB, free: 901.8 MB)
18/06/20 11:36:23 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1039
18/06/20 11:36:23 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[15] at count at SimpleApp.java:16) (first 15 tasks are for partitions Vector(0))
18/06/20 11:36:23 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
18/06/20 11:36:23 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, executor driver, partition 0, PROCESS_LOCAL, 8316 bytes)
18/06/20 11:36:23 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
18/06/20 11:36:23 INFO BlockManager: Found block rdd_2_0 locally
18/06/20 11:36:23 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1898 bytes result sent to driver
18/06/20 11:36:23 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 35 ms on localhost (executor driver) (1/1)
18/06/20 11:36:23 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
18/06/20 11:36:23 INFO DAGScheduler: ShuffleMapStage 2 (count at SimpleApp.java:16) finished in 0.042 s
18/06/20 11:36:23 INFO DAGScheduler: looking for newly runnable stages
18/06/20 11:36:23 INFO DAGScheduler: running: Set()
18/06/20 11:36:23 INFO DAGScheduler: waiting: Set(ResultStage 3)
18/06/20 11:36:23 INFO DAGScheduler: failed: Set()
18/06/20 11:36:23 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[18] at count at SimpleApp.java:16), which has no missing parents
18/06/20 11:36:23 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 7.4 KB, free 901.5 MB)
18/06/20 11:36:23 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 3.8 KB, free 901.5 MB)
18/06/20 11:36:23 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on yutao.go-goal.com:49470 (size: 3.8 KB, free: 901.8 MB)
18/06/20 11:36:23 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1039
18/06/20 11:36:23 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[18] at count at SimpleApp.java:16) (first 15 tasks are for partitions Vector(0))
18/06/20 11:36:23 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
18/06/20 11:36:23 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, executor driver, partition 0, ANY, 7754 bytes)
18/06/20 11:36:23 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
18/06/20 11:36:23 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
18/06/20 11:36:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
18/06/20 11:36:23 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1696 bytes result sent to driver
18/06/20 11:36:23 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 6 ms on localhost (executor driver) (1/1)
18/06/20 11:36:23 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
18/06/20 11:36:23 INFO DAGScheduler: ResultStage 3 (count at SimpleApp.java:16) finished in 0.013 s
18/06/20 11:36:23 INFO DAGScheduler: Job 1 finished: count at SimpleApp.java:16, took 0.060497 s
// 重点看这里 执行成功了
Lines with a:61, lines with b: 30
18/06/20 11:36:23 INFO SparkUI: Stopped Spark web UI at http://yutao.go-goal.com:4040
18/06/20 11:36:23 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
18/06/20 11:36:23 INFO MemoryStore: MemoryStore cleared
18/06/20 11:36:23 INFO BlockManager: BlockManager stopped
18/06/20 11:36:23 INFO BlockManagerMaster: BlockManagerMaster stopped
18/06/20 11:36:23 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
18/06/20 11:36:23 INFO SparkContext: Successfully stopped SparkContext
18/06/20 11:36:23 INFO ShutdownHookManager: Shutdown hook called
18/06/20 11:36:23 INFO ShutdownHookManager: Deleting directory C:\Users\yutao\AppData\Local\Temp\spark-48b4a38a-c25a-4da7-b2fc-de0da9b7609b