spark WARNTaskSchedulerImpl:Initial job has not accepted any resources; check your cluster UI to

spark在提交任务时，出现如下错误：

15/03/26 22:29:36 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/03/26 22:29:51 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/03/26 22:30:06 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/03/26 22:30:21 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

从警告信息上看，初始化job时没有获取到任何资源；提示检查集群，确保workers可以被注册并有足够的内存资源。

如上问题产生的原因是多方面的，可能原因如下：

1.因为提交任务的节点不能和spark工作节点交互，因为提交完任务后提交任务节点上会起一个进程，展示任务进度，大多端口为4044，工作节点需要反馈进度给该该端口，所以如果主机名或者IP在hosts中配置不正确。所以检查下主机名和ip是否配置正确

2.有可能是内存不足

  检查内存

   conf.set("spark.executor.memory", "3000m")

  Make sure to set SPARK_LOCAL_IP andSPARK_MASTER_IP.

  查看8080端口，确保一些workers保持Alive状态，确保 some cores 是可利用的。

 

3.driver在不停地工作，一直处于running状态，而从前端8080页面又可看出程序的状态一直是WAITING。submit所在的shell输出driver信息后便退出了，目前还未得到程序运行结果，这显然是因为该程序处于WAITING状态，还未运行。而导致这一结果，应该跟日志的输出有关。值得一提的是，到目前为止，出了更改了对master ip和port的配置以及worker的配置外，其他深入的配置还需进一步研究。