虽然报异常了
但依然出来结果了
emmm
1. SparkException: Could not find CoarseGrainedScheduler.
具体报错信息
19/08/23 11:13:58 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/08/23 11:13:59 ERROR Utils: Uncaught exception in thread driver-revive-thread
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:154)
at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:134)
at org.apache.spark.rpc.netty.NettyRpcEnv.send(NettyRpcEnv.scala:186)
at org.apache.spark.rpc.netty.NettyRpcEndpointRef.send(NettyRpcEnv.scala:517)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(CoarseGrainedSchedulerBackend.scala:116)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(CoarseGrainedSchedulerBackend.scala:116)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anon$1$$anonfun$run$1.apply$mcV$sp(CoarseGrainedSchedulerBackend.scala:116)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1317)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anon$1.run(CoarseGrainedSchedulerBackend.scala:115)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/08/23 11:14:03 INFO MemoryStore: MemoryStore cleared
19/08/23 11:14:03 INFO BlockManager: BlockManager stopped
19/08/23 11:14:03 INFO BlockManagerMaster: BlockManagerMaster stopped
19/08/23 11:14:03 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/08/23 11:14:04 INFO SparkContext: Successfully stopped SparkContext
19/08/23 11:14:05 INFO ShutdownHookManager: Shutdown hook called
19/08/23 11:14:05 INFO ShutdownHookManager: Deleting directory /tmp/spark-3f6a96e5-4ad2-4080-8245-cb73569cfcc7
1、这个可能是一个资源问题,应该给任务分配更多的 cores 和Executors,并且分配更多的内存。并且需要给RDD分配更多的分区
2、在配置资源中加入这句话也许能解决你的问题:–conf spark.dynamicAllocation.enabled=false
看到这个博客 Spark 常见问题解决方案
依照第一个方法
解决了问题
不过两次结果都是一样的 这可能是普通的异常吧
[root@hadoop01 bin]# ./spark-submit \
--class SparkWordCount \
--master spark://hadoop01:7077 \
--executor-memory 1G \
--total-executor-cores 2 \
/home/sparkCore-1.0-SNAPSHOT.jar hdfs://hadoop01:8020/wordcount hdfs://hadoop01:8020/out1
我就换成了2和4就没问题了
[root@hadoop01 bin]# ./spark-submit \
--class SparkWordCount \
--master spark://hadoop01:7077 \
--executor-memory 2G \
--total-executor-cores 4 \
/home/sparkCore-1.0-SNAPSHOT.jar hdfs://hadoop01:8020/wordcount hdfs://hadoop01:8020/out2