Executor updated: app-20181031153851-0007/1 is now EXITED (Command exited with code 1)

最新推荐文章于 2023-03-18 08:41:21 发布
最新推荐文章于 2023-03-18 08:41:21 发布
阅读量2.1k 收藏 2
点赞数 1
分类专栏： Spark 文章标签： Executor updated  is now EXITED Command exited with code 1
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/huanbia/article/details/83586695
版权

最近在尝试通过python和java在本地远程调用spark，具体如何用java远程调用spark，可参考我另一篇博客：
Idea基于maven，java语言的spark环境搭建

实验环境

client集群
centos单机4个hadoop节点集群 ，ip分别为192.168.20.[61,62,63,64]，61为主节点

问题描述

代码其实比较简单，java代码如下，主要意思是访问我的hdfs上的文件，并统计行数。

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;

public class JavaTest {
    public static final String master = "spark://192.168.20.61:7077";
    public static void main(String[] args) {
        SparkConf conf = new SparkConf().setAppName("demo").setMaster(master);
        JavaSparkContext sc = new JavaSparkContext(conf);
        System.out.println(sc.textFile("hdfs://192.168.20.61:9000/flume.exec.log").count());
        sc.stop();
    }
}

运行时报一些错误，其中循环的比较多，我摘出主要错误信息如下，完整错误信息可查看附录。这个错误信息大致意思是，Executor成功添加后，然后就会down掉，进而继续添加，再down掉，再添加这种反复的过程。

18/10/31 15:38:07 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/3 on worker-20181029153030-192.168.20.63-48117 (192.168.20.63:48117) with 8 cores
18/10/31 15:38:07 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/3 on hostPort 192.168.20.63:48117 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:07 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/3 is now RUNNING
18/10/31 15:38:08 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/0 is now EXITED (Command exited with code 1)
18/10/31 15:38:08 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/0 removed: Command exited with code 1
18/10/31 15:38:08 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 0
18/10/31 15:38:08 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/4 on worker-20181029153028-192.168.20.62-36690 (192.168.20.62:36690) with 8 cores
18/10/31 15:38:08 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/4 on hostPort 192.168.20.62:36690 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:08 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/4 is now RUNNING
18/10/31 15:38:08 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/2 is now EXITED (Command exited with code 1)
18/10/31 15:38:08 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/2 removed: Command exited with code 1
18/10/31 15:38:08 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 2

我的解决方案

考虑到通信问题，既然是本地来获得集群的executor状态，有可能是自己无法接受集群传递过来的状态，可考虑将本地的防火墙关掉，尝试看看是否能解决问题，我的问题就是由于本地没有关闭防火墙导致的。
关掉防火墙命令如下：

su root
service iptables stop

后记

自己当时查了很多资料，有各种各样的解决方案，有说java无法执行的，有说数据量太大的，我遇到的只是通信问题，大家可以尝试一下，其余解决方案连接如下：

  1. https://blog.csdn.net/qianlong4526888/article/details/22899355
  2. https://bbs.csdn.net/topics/390970634

附录

  • 错误信息完整代码
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
18/10/31 15:38:04 INFO SparkContext: Running Spark version 1.6.1
18/10/31 15:38:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/10/31 15:38:05 INFO SecurityManager: Changing view acls to: hadoop
18/10/31 15:38:05 INFO SecurityManager: Changing modify acls to: hadoop
18/10/31 15:38:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
18/10/31 15:38:05 INFO Utils: Successfully started service 'sparkDriver' on port 43676.
18/10/31 15:38:05 INFO Slf4jLogger: Slf4jLogger started
18/10/31 15:38:05 INFO Remoting: Starting remoting
18/10/31 15:38:05 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.20.210:60649]
18/10/31 15:38:05 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 60649.
18/10/31 15:38:05 INFO SparkEnv: Registering MapOutputTracker
18/10/31 15:38:05 INFO SparkEnv: Registering BlockManagerMaster
18/10/31 15:38:05 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-5e7f6678-df22-4f4a-aeec-dd642cba5ad0
18/10/31 15:38:05 INFO MemoryStore: MemoryStore started with capacity 1077.8 MB
18/10/31 15:38:05 INFO SparkEnv: Registering OutputCommitCoordinator
18/10/31 15:38:05 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/10/31 15:38:05 INFO SparkUI: Started SparkUI at http://192.168.20.210:4040
18/10/31 15:38:06 INFO AppClient$ClientEndpoint: Connecting to master spark://192.168.20.61:7077...
18/10/31 15:38:06 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20181031153851-0007
18/10/31 15:38:06 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/0 on worker-20181029153028-192.168.20.62-36690 (192.168.20.62:36690) with 8 cores
18/10/31 15:38:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/0 on hostPort 192.168.20.62:36690 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:06 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/1 on worker-20181029153030-192.168.20.63-48117 (192.168.20.63:48117) with 8 cores
18/10/31 15:38:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/1 on hostPort 192.168.20.63:48117 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:06 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/2 on worker-20181029153036-192.168.20.64-52254 (192.168.20.64:52254) with 8 cores
18/10/31 15:38:06 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/2 on hostPort 192.168.20.64:52254 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:06 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54421.
18/10/31 15:38:06 INFO NettyBlockTransferService: Server created on 54421
18/10/31 15:38:06 INFO BlockManagerMaster: Trying to register BlockManager
18/10/31 15:38:06 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.20.210:54421 with 1077.8 MB RAM, BlockManagerId(driver, 192.168.20.210, 54421)
18/10/31 15:38:06 INFO BlockManagerMaster: Registered BlockManager
18/10/31 15:38:06 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/1 is now RUNNING
18/10/31 15:38:06 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/0 is now RUNNING
18/10/31 15:38:06 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/2 is now RUNNING
18/10/31 15:38:06 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
18/10/31 15:38:06 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 127.0 KB, free 127.0 KB)
18/10/31 15:38:06 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 15.5 KB, free 142.5 KB)
18/10/31 15:38:06 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.20.210:54421 (size: 15.5 KB, free: 1077.7 MB)
18/10/31 15:38:06 INFO SparkContext: Created broadcast 0 from textFile at JavaTest.java:9
18/10/31 15:38:06 INFO FileInputFormat: Total input paths to process : 1
18/10/31 15:38:06 INFO SparkContext: Starting job: count at JavaTest.java:9
18/10/31 15:38:06 INFO DAGScheduler: Got job 0 (count at JavaTest.java:9) with 2 output partitions
18/10/31 15:38:06 INFO DAGScheduler: Final stage: ResultStage 0 (count at JavaTest.java:9)
18/10/31 15:38:06 INFO DAGScheduler: Parents of final stage: List()
18/10/31 15:38:06 INFO DAGScheduler: Missing parents: List()
18/10/31 15:38:06 INFO DAGScheduler: Submitting ResultStage 0 (hdfs://192.168.20.61:9000/flume.exec.log MapPartitionsRDD[1] at textFile at JavaTest.java:9), which has no missing parents
18/10/31 15:38:06 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.9 KB, free 145.4 KB)
18/10/31 15:38:06 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1798.0 B, free 147.2 KB)
18/10/31 15:38:06 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.20.210:54421 (size: 1798.0 B, free: 1077.7 MB)
18/10/31 15:38:06 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
18/10/31 15:38:06 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (hdfs://192.168.20.61:9000/flume.exec.log MapPartitionsRDD[1] at textFile at JavaTest.java:9)
18/10/31 15:38:06 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
18/10/31 15:38:07 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/1 is now EXITED (Command exited with code 1)
18/10/31 15:38:07 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/1 removed: Command exited with code 1
18/10/31 15:38:07 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 1
18/10/31 15:38:07 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/3 on worker-20181029153030-192.168.20.63-48117 (192.168.20.63:48117) with 8 cores
18/10/31 15:38:07 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/3 on hostPort 192.168.20.63:48117 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:07 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/3 is now RUNNING
18/10/31 15:38:08 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/0 is now EXITED (Command exited with code 1)
18/10/31 15:38:08 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/0 removed: Command exited with code 1
18/10/31 15:38:08 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 0
18/10/31 15:38:08 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/4 on worker-20181029153028-192.168.20.62-36690 (192.168.20.62:36690) with 8 cores
18/10/31 15:38:08 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/4 on hostPort 192.168.20.62:36690 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:08 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/4 is now RUNNING
18/10/31 15:38:08 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/2 is now EXITED (Command exited with code 1)
18/10/31 15:38:08 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/2 removed: Command exited with code 1
18/10/31 15:38:08 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 2
18/10/31 15:38:08 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/5 on worker-20181029153036-192.168.20.64-52254 (192.168.20.64:52254) with 8 cores
18/10/31 15:38:08 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/5 on hostPort 192.168.20.64:52254 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:08 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/5 is now RUNNING
18/10/31 15:38:09 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/3 is now EXITED (Command exited with code 1)
18/10/31 15:38:09 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/3 removed: Command exited with code 1
18/10/31 15:38:09 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 3
18/10/31 15:38:09 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/6 on worker-20181029153030-192.168.20.63-48117 (192.168.20.63:48117) with 8 cores
18/10/31 15:38:09 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/6 on hostPort 192.168.20.63:48117 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:09 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/6 is now RUNNING
18/10/31 15:38:09 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/4 is now EXITED (Command exited with code 1)
18/10/31 15:38:09 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/4 removed: Command exited with code 1
18/10/31 15:38:09 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 4
18/10/31 15:38:09 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/7 on worker-20181029153028-192.168.20.62-36690 (192.168.20.62:36690) with 8 cores
18/10/31 15:38:09 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/7 on hostPort 192.168.20.62:36690 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:09 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/7 is now RUNNING
18/10/31 15:38:09 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/5 is now EXITED (Command exited with code 1)
18/10/31 15:38:09 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/5 removed: Command exited with code 1
18/10/31 15:38:09 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 5
18/10/31 15:38:09 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/8 on worker-20181029153036-192.168.20.64-52254 (192.168.20.64:52254) with 8 cores
18/10/31 15:38:09 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/8 on hostPort 192.168.20.64:52254 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:09 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/8 is now RUNNING
18/10/31 15:38:11 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/7 is now EXITED (Command exited with code 1)
18/10/31 15:38:11 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/7 removed: Command exited with code 1
18/10/31 15:38:11 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 7
18/10/31 15:38:11 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/9 on worker-20181029153028-192.168.20.62-36690 (192.168.20.62:36690) with 8 cores
18/10/31 15:38:11 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/9 on hostPort 192.168.20.62:36690 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:11 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/9 is now RUNNING
18/10/31 15:38:14 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/6 is now EXITED (Command exited with code 1)
18/10/31 15:38:14 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/6 removed: Command exited with code 1
18/10/31 15:38:14 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 6
18/10/31 15:38:14 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/10 on worker-20181029153030-192.168.20.63-48117 (192.168.20.63:48117) with 8 cores
18/10/31 15:38:14 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/10 on hostPort 192.168.20.63:48117 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:14 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/10 is now RUNNING
18/10/31 15:38:14 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/8 is now EXITED (Command exited with code 1)
18/10/31 15:38:14 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/8 removed: Command exited with code 1
18/10/31 15:38:14 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 8
18/10/31 15:38:14 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/11 on worker-20181029153036-192.168.20.64-52254 (192.168.20.64:52254) with 8 cores
18/10/31 15:38:14 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/11 on hostPort 192.168.20.64:52254 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:14 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/11 is now RUNNING
18/10/31 15:38:16 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/9 is now EXITED (Command exited with code 1)
18/10/31 15:38:16 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/9 removed: Command exited with code 1
18/10/31 15:38:16 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 9
18/10/31 15:38:16 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/12 on worker-20181029153028-192.168.20.62-36690 (192.168.20.62:36690) with 8 cores
18/10/31 15:38:16 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/12 on hostPort 192.168.20.62:36690 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:16 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/12 is now RUNNING
18/10/31 15:38:19 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/10 is now EXITED (Command exited with code 1)
18/10/31 15:38:19 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/10 removed: Command exited with code 1
18/10/31 15:38:19 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 10
18/10/31 15:38:19 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/13 on worker-20181029153030-192.168.20.63-48117 (192.168.20.63:48117) with 8 cores
18/10/31 15:38:19 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/13 on hostPort 192.168.20.63:48117 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:19 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/13 is now RUNNING
18/10/31 15:38:19 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/11 is now EXITED (Command exited with code 1)
18/10/31 15:38:19 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/11 removed: Command exited with code 1
18/10/31 15:38:19 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 11
18/10/31 15:38:19 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/14 on worker-20181029153036-192.168.20.64-52254 (192.168.20.64:52254) with 8 cores
18/10/31 15:38:19 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/14 on hostPort 192.168.20.64:52254 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:19 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/14 is now RUNNING
18/10/31 15:38:21 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/12 is now EXITED (Command exited with code 1)
18/10/31 15:38:21 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/12 removed: Command exited with code 1
18/10/31 15:38:21 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 12
18/10/31 15:38:21 INFO AppClient$ClientEndpoint: Executor added: app-20181031153851-0007/15 on worker-20181029153028-192.168.20.62-36690 (192.168.20.62:36690) with 8 cores
18/10/31 15:38:21 INFO SparkDeploySchedulerBackend: Granted executor ID app-20181031153851-0007/15 on hostPort 192.168.20.62:36690 with 8 cores, 1024.0 MB RAM
18/10/31 15:38:21 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/15 is now RUNNING
18/10/31 15:38:21 INFO AppClient$ClientEndpoint: Executor updated: app-20181031153851-0007/13 is now EXITED (Command exited with code 1)
18/10/31 15:38:21 INFO SparkDeploySchedulerBackend: Executor app-20181031153851-0007/13 removed: Command exited with code 1
18/10/31 15:38:21 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 13
huanbia
关注
  • 1
    点赞
  • 踩
  • 2
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
专栏目录
"spark is now FAILED (Command exited with code 1)" 错误解决
egraldloi的专栏
03-08 4974
今天配置spark开发环境，出现了一个问题，这个问题网上有人问到，但是没有查询到解决方法，最后问题得以解决。 出现的问题是，当启动一个集群的spark-shell的时候，出现了： "spark is now FAILED (Command exited with code 1)"，这个问题的出现伴随着executor节点启动后马上又和master失去了联系，如下： 14/03/08 10:5
Spark2.2 Driver和Executor状态改变处理机制源码分析
行走的树
10-23 637
Driver状态发生改变处理机制 DriverStateChanged( ) Executor状态发生改变处理机制 ExecutorStateChanged( )
Spark Executor模块详解
晓晓的天空
03-18 1728
Spark Executor模块详解
Delphi Android 开发中 [Exec Error] The command ...exited with code 1 错误的解决
newsxy的专栏
03-17 2035
使用Lite版全新安装了Delphi 10.3 Rio，Windows开发一切正常，在进行Android开发时无法进行Build，新建的空白Multi-Device Application也无法Build，出现同样的错误提示 因为计算机中之前安装了IntelliJ IDEA 、Eclipse Java、Tomcat等Java开发环境，加上对Android开发不了解，认为是Java环境、SDK...
使用IDEA 构建入门级程序以及问题解决
tu5213xf的专栏
01-08 2425
学习spark                                   配置环境变量        vim /etc/profile 的配置     export JAVA_HOME=/usr/java/jdk1.7.0_60     export HADOOP_HOME=/itcast/hadoop-2.2.0     export a
xxl-job-executor:与xxl-job-executor的集成
04-19
xxl-job-executor的gin中间件背景xxl-job-executor-go是xxl-job的golang执行器，可以独立运行，有时候我们要与项目或者框架（如:gin框架）集成起来合并为一个服务，本项目因此而生。执行器项目地址与gin集成示例...
MyBatis-Simplified-Chinese.rar_ mybatis-3_Mybatis-spring_mybatis
09-20
3. Executor 执行器：负责执行 SQL 映射文件中的 SQL 语句。 4. ParameterHandler 参数处理器：处理输入参数，将 Java 对象转换为 SQL 执行所需的格式。 5. ResultHandler 结果处理器：处理 SQL 查询的结果，将结果...
executor-check-online-service:待命模块
05-04
流程管理的新方法的在线检查执行：安装： 通过NPM npm i @runnerty/executor-check-online-service 您还可以使用将模块添加到项目中npx runnerty-cli add @runnerty/executor-check-online-service 此命令在您的项目...
大数据组件-监控-spark-driver/executor性能的prometheus-grafana模板插件
01-29
大数据组件-监控-spark-driver/executor性能的prometheus-grafana模板插件
executor-puppeteer-export:待命模块
05-08
npm i @runnerty/executor-puppeteer-export 您还可以使用将模块添加到项目中 npx runnerty-cli add @runnerty/executor-puppeteer-export 此命令在您的项目中安装模块，在config.json添加示例配置，并创建示例...
windows环境下IDEA运行spark程序出现的异常问题
philonyun的博客
05-19 2918
WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
spark2.11-2.3源码4_Master状态改变
u011607686的博客
01-17 161
DriverStateChanged case DriverStateChanged(driverId, state, exception) => state match { //如果driver状态是ERROR\FINISHED\KILLED\FAILED ,removeDriver case DriverState.ERR...
Spark调优之旅<一>
李木子兮的博客
10-13 717
问题说明 1：spark-shell启动时重复提示：executor updated:app-2016*** is now Running                                                     executor updated:app-2016*** is now Loading                executor update
Web提交参数到Spark集群执行任务
nanphonfy(南风)
09-03 7369
提供一个API供使用者调用，大抵需求如下： 输入某用户的位置（经纬度），提交到Web服务器，再把任务（找出该用户附近一公里内的商城推荐）提交到Spark集群上执行，返回计算结果后再存入到redis数据库中，供后台调用返回结果给使用方。 网上关于这方面的资料大抵是基于spark-assembly-1.4.1-hadoop2.6.0.jar，而我们这边的环境是spark2.0-Hadoop2.6.0
spark常见问题汇总
JXL_1207的博客
10-21 838
转载自：https://www.cnblogs.com/arachis/p/Spark_Exception.html 写的很全面，值得借鉴。 原文地址：https://my.oschina.net/tearsky/blog/629201 摘要： 　　1、Operation category READ is not supported in state standby 　　2、配置spar...
Spark SQL CLI 运行
热门推荐
风行天下
03-04 1万+
1：运行 ./bin/spark-sql 需要先把hive-site.xml 负责到spark的conf目录下 [jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./bin/spark-sql Spark assembly has been built with Hive, including Datanucleus jars on classpath java.l
IOS+Appium自动化遇到的问题汇总
yinshuilan的专栏
06-06 2129
python第三方库pymssql无法使用 （因重新安装brew，python等应用后导致，重装后可以使用） freetds源码下载地址： ftp://ftp.freetds.org/pub/freetds/stable/ pymssql源码下载地址： https://pypi.python.org/pypi/pymssql/ 先用 brew list 查看已经安装的包 如果已经安装...
第1关:spark-submit提交 解析
最新发布
04-05
Spark-submit是Spark自带的提交脚本，用于将Spark应用程序提交到集群中运行。Spark-submit可以将应用程序打包成一个JAR包并提交到集群中运行，也可以直接提交一个Python文件或者一个Scala文件。 使用Spark-submit提交应用程序时，需要指定以下参数： 1. --class：指定主类名，如果是Java应用程序，需要指定该参数；如果是Scala应用程序，可以省略该参数，Spark-submit会自动查找Scala文件中的main函数。 2. --master：指定运行模式，可以是local、yarn、mesos等。 3. --deploy-mode：指定部署模式，可以是client或者cluster，如果是client模式，则Driver运行在提交任务的机器上；如果是cluster模式，则Driver运行在集群中的某个节点上。 4. --executor-memory：指定Executor的内存大小。 5. --total-executor-cores：指定Executor的总核数。 6. --num-executors：指定Executor的个数。 7. 应用程序的JAR包路径或者Python/Scala文件路径。 例如，使用Spark-submit提交一个Java应用程序，命令如下： ``` ./bin/spark-submit --class com.spark.example.WordCount --master yarn --deploy-mode client --executor-memory 2g --total-executor-cores 4 --num-executors 2 /path/to/WordCount.jar /path/to/input /path/to/output ``` 其中，--class指定了Java应用程序的主类名为com.spark.example.WordCount，--master指定了运行模式为yarn，--deploy-mode指定了部署模式为client，--executor-memory指定了每个Executor的内存大小为2g，--total-executor-cores指定了Executor总核数为4，--num-executors指定了Executor的个数为2，最后两个参数为输入和输出路径。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值