Spark 关于提交任务报错 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources;

运行任务发现一直在报:WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

查看7077端口发现我们要执行的任务处于等待状态,当前正在运行的是shell,所有我们直接在webui kill 掉 shell

kill 掉后任务重新执行了

22/04/12 22:03:48 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:04:03 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:04:18 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:04:33 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:04:48 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:05:03 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:05:18 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:05:33 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:05:48 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:06:03 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:06:18 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:06:33 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:06:48 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:07:03 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:07:18 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/04/12 22:07:19 INFO client.StandaloneAppClient$ClientEndpoint: Executor added: app-20220412215715-0004/0 on worker-20220412152324-192.3.43.112-41110 (192.3.43.112:41110) with 1 core(s)
22/04/12 22:07:19 INFO cluster.StandaloneSchedulerBackend: Granted executor ID app-20220412215715-0004/0 on hostPort 192.3.43.112:41110 with 1 core(s), 512.0 MB RAM
22/04/12 22:07:19 INFO client.StandaloneAppClient$ClientEndpoint: Executor updated: app-20220412215715-0004/0 is now RUNNING
22/04/12 22:07:19 INFO client.StandaloneAppClient$ClientEndpoint: Executor added: app-20220412215715-0004/1 on worker-20220412152325-192.3.43.111-33615 (192.3.43.111:33615) with 1 core(s)
22/04/12 22:07:19 INFO cluster.StandaloneSchedulerBackend: Granted executor ID app-20220412215715-0004/1 on hostPort 192.3.43.111:33615 with 1 core(s), 512.0 MB RAM
22/04/12 22:07:19 INFO client.StandaloneAppClient$ClientEndpoint: Executor updated: app-20220412215715-0004/1 is now RUNNING
22/04/12 22:07:23 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.3.43.112:52246) with ID 0
22/04/12 22:07:23 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 192.3.43.112, executor 0, partition 0, PROCESS_LOCAL, 7870 bytes)
22/04/12 22:07:23 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.3.43.111:58412) with ID 1
22/04/12 22:07:23 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 192.3.43.111, executor 1, partition 1, PROCESS_LOCAL, 7870 bytes)
22/04/12 22:07:24 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.3.43.112:45912 with 117.0 MB RAM, BlockManagerId(0, 192.3.43.112, 45912, None)
22/04/12 22:07:24 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.3.43.111:32810 with 117.0 MB RAM, BlockManagerId(1, 192.3.43.111, 32810, None)
22/04/12 22:07:26 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.3.43.112:45912 (size: 1256.0 B, free: 117.0 MB)
22/04/12 22:07:26 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.3.43.111:32810 (size: 1256.0 B, free: 117.0 MB)
22/04/12 22:07:27 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3266 ms on 192.3.43.112 (executor 0) (1/2)
22/04/12 22:07:27 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 3613 ms on 192.3.43.111 (executor 1) (2/2)
22/04/12 22:07:27 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
22/04/12 22:07:27 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 609.160 s
22/04/12 22:07:27 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 609.382372 s
Pi is roughly 3.138155690778454
22/04/12 22:07:27 INFO server.AbstractConnector: Stopped Spark@2375b321{HTTP/1.1,[http/1.1]}{0.0.0.0:4041}
22/04/12 22:07:27 INFO ui.SparkUI: Stopped Spark web UI at http://Mastertjy43:4041
22/04/12 22:07:27 INFO cluster.StandaloneSchedulerBackend: Shutting down all executors
22/04/12 22:07:27 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
22/04/12 22:07:27 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/04/12 22:07:27 INFO memory.MemoryStore: MemoryStore cleared
22/04/12 22:07:27 INFO storage.BlockManager: BlockManager stopped
22/04/12 22:07:27 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
22/04/12 22:07:27 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/04/12 22:07:27 INFO spark.SparkContext: Successfully stopped SparkContext
22/04/12 22:07:27 INFO util.ShutdownHookManager: Shutdown hook called
22/04/12 22:07:27 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-ea10ab9c-e9ae-4f4d-82e3-7b4fd78610c7
22/04/12 22:07:27 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-b2a5d784-9dee-45ea-8c3a-128f97dc9228

  • 2
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值