Spark问题Could not find CoarseGrainedScheduler

虽然报异常了
但依然出来结果了
emmm

1. SparkException: Could not find CoarseGrainedScheduler.

具体报错信息

19/08/23 11:13:58 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/08/23 11:13:59 ERROR Utils: Uncaught exception in thread driver-revive-thread
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
	at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:154)
	at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:134)
	at org.apache.spark.rpc.netty.NettyRpcEnv.send(NettyRpcEnv.scala:186)
	at org.apache.spark.rpc.netty.NettyRpcEndpointRef.send(NettyRpcEnv.scala:517)
	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(CoarseGrainedSchedulerBackend.scala:116)
	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(CoarseGrainedSchedulerBackend.scala:116)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anon$1$$anonfun$run$1.apply$mcV$sp(CoarseGrainedSchedulerBackend.scala:116)
	at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1317)
	at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverEndpoint$$anon$1.run(CoarseGrainedSchedulerBackend.scala:115)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
19/08/23 11:14:03 INFO MemoryStore: MemoryStore cleared
19/08/23 11:14:03 INFO BlockManager: BlockManager stopped
19/08/23 11:14:03 INFO BlockManagerMaster: BlockManagerMaster stopped
19/08/23 11:14:03 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/08/23 11:14:04 INFO SparkContext: Successfully stopped SparkContext
19/08/23 11:14:05 INFO ShutdownHookManager: Shutdown hook called
19/08/23 11:14:05 INFO ShutdownHookManager: Deleting directory /tmp/spark-3f6a96e5-4ad2-4080-8245-cb73569cfcc7

1、这个可能是一个资源问题，应该给任务分配更多的 cores 和Executors，并且分配更多的内存。并且需要给RDD分配更多的分区
2、在配置资源中加入这句话也许能解决你的问题：–conf spark.dynamicAllocation.enabled=false

看到这个博客 Spark 常见问题解决方案
依照第一个方法
解决了问题

不过两次结果都是一样的 这可能是普通的异常吧

[root@hadoop01 bin]# ./spark-submit \
--class SparkWordCount \
--master spark://hadoop01:7077 \
--executor-memory 1G \
--total-executor-cores 2 \
/home/sparkCore-1.0-SNAPSHOT.jar hdfs://hadoop01:8020/wordcount hdfs://hadoop01:8020/out1

我就换成了2和4就没问题了

[root@hadoop01 bin]# ./spark-submit \
--class SparkWordCount \
--master spark://hadoop01:7077 \
--executor-memory 2G \
--total-executor-cores 4 \
/home/sparkCore-1.0-SNAPSHOT.jar hdfs://hadoop01:8020/wordcount hdfs://hadoop01:8020/out2