spark 解决办法 check your cluster UI to ensure that workers are registered and have sufficient memory

最新推荐文章于 2022-11-09 00:35:49 发布

wyx100

最新推荐文章于 2022-11-09 00:35:49 发布

阅读量8k

点赞数 1

文章标签： spark submit

本文链接：https://blog.csdn.net/wyx100/article/details/51871597

版权

报错：WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

原因：内存不足

解决办法：设置内存

spark-submit --master spark://eb174:7077 --name WordCountByscala --class com.hq.WordCount --executor-memory 1G --total-executor-cores 2 ~/test/WordCount.jar hdfs://eb170:8020/user/ebupt/text

修改为

spark-submit --master spark://eb174:7077 --name WordCountByscala --class com.hq.WordCount --executor-memory 512M --total-executor-cores 2 ~/test/WordCount.jar hdfs://eb170:8020/user/ebupt/text

spark集群运行

 1 package com.hq
 2 
 3 /**
 4  * User: hadoop
 5  * Date: 2014/10/10 0010
 6  * Time: 18:59
 7  */
 8 import org.apache.spark.SparkConf
 9 import org.apache.spark.SparkContext
10 import org.apache.spark.SparkContext._
11 
12 /**
13  * 统计字符出现次数
14  */
15 object WordCount {
16   def main(args: Array[String]) {
17     if (args.length < 1) {
18       System.err.println("Usage: <file>")
19       System.exit(1)
20     }
21 
22     val conf = new SparkConf()
23     val sc = new SparkContext(conf)
24     val line = sc.textFile(args(0))
25 
26     line.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_).collect().foreach(println)
27 
28     sc.stop()
29   }
30 }

执行：

spark-submit --master spark://eb174:7077 --name WordCountByscala --class com.hq.WordCount --executor-memory 1G --total-executor-cores 2 ~/test/WordCount.jar hdfs://eb170:8020/user/ebupt/text

报错：

WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory