【解决】WARN YarnClusterSch…

最新推荐文章于 2022-11-05 15:51:50 发布
最新推荐文章于 2022-11-05 15:51:50 发布
阅读量1.1k 收藏
点赞数
分类专栏： 【C、Java与网络编程】
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u010770993/article/details/70312485
版权
这篇博客记录了解决Spark在Yarn集群上运行时遇到的WARN YarnClusterScheduler警告的过程。通过调整配置和检查资源分配，成功避免了初始作业未接受任何资源的问题。在解决问题后，观察到正常日志输出，包括Spark版本、安全设置、资源管理器注册等，显示应用程序已正常启动并运行。
摘要由CSDN通过智能技术生成

解决 WARN YarnClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
后正常的日志：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoopTmp/nm-local-dir/filecache/148/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/02/26 23:59:50 INFO ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
17/02/26 23:59:51 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1488181659296_0004_000001
17/02/26 23:59:52 INFO SecurityManager: Changing view acls to: hadoop
17/02/26 23:59:52 INFO SecurityManager: Changing modify acls to: hadoop
17/02/26 23:59:52 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
17/02/26 23:59:52 INFO ApplicationMaster: Starting the user application in a separate Thread
17/02/26 23:59:52 INFO ApplicationMaster: Waiting for spark context initialization
17/02/26 23:59:52 INFO ApplicationMaster: Waiting for spark context initialization ... 
17/02/26 23:59:53 INFO SparkContext: Running Spark version 1.6.1
17/02/26 23:59:53 INFO SecurityManager: Changing view acls to: hadoop
17/02/26 23:59:53 INFO SecurityManager: Changing modify acls to: hadoop
17/02/26 23:59:53 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
17/02/26 23:59:53 INFO Utils: Successfully started service 'sparkDriver' on port 46344.
17/02/26 23:59:53 INFO Slf4jLogger: Slf4jLogger started
17/02/26 23:59:53 INFO Remoting: Starting remoting
17/02/26 23:59:53 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.136.135:46211]
17/02/26 23:59:53 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 46211.
17/02/26 23:59:53 INFO SparkEnv: Registering MapOutputTracker
17/02/26 23:59:53 INFO SparkEnv: Registering BlockManagerMaster
17/02/26 23:59:53 INFO DiskBlockManager: Created local directory at /home/hadoop/hadoopTmp/nm-local-dir/usercache/hadoop/appcache/application_1488181659296_0004/blockmgr-93703b9e-2d79-4494-a02d-7dfdb349da89
17/02/26 23:59:53 INFO MemoryStore: MemoryStore started with capacity 457.9 MB
17/02/26 23:59:53 INFO SparkEnv: Registering OutputCommitCoordinator
17/02/26 23:59:54 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/02/26 23:59:54 INFO Utils: Successfully started service 'SparkUI' on port 35252.
17/02/26 23:59:54 INFO SparkUI: Started SparkUI at http://192.168.136.135:35252
17/02/26 23:59:54 INFO YarnClusterScheduler: Created YarnClusterScheduler
17/02/26 23:59:54 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43620.
17/02/26 23:59:54 INFO NettyBlockTransferService: Server created on 43620
17/02/26 23:59:54 INFO BlockManagerMaster: Trying to register BlockManager
17/02/26 23:59:54 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.136.135:43620 with 457.9 MB RAM, BlockManagerId(driver, 192.168.136.135, 43620)
17/02/26 23:59:54 INFO BlockManagerMaster: Registered BlockManager
17/02/26 23:59:54 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@192.168.136.135:46344)
17/02/26 23:59:54 INFO RMProxy: Connecting to ResourceManager at bigmaster/192.168.136.134:8030
17/02/26 23:59:54 INFO YarnRMClient: Registering the ApplicationMaster
17/02/26 23:59:55 INFO YarnAllocator: Will request 2 executor containers, each with 1 cores and 1408 MB memory including 384 MB overhead
17/02/26 23:59:55 INFO YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
17/02/26 23:59:55 INFO YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
17/02/26 23:59:55 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
17/02/26 23:59:55 INFO AMRMClientImpl: Received new token for : big1:38108
17/02/26 23:59:55 INFO YarnAllocator: Launching container container_1488181659296_0004_01_000002 for on host big1
17/02/26 23:59:55 INFO YarnAllocator: Launching ExecutorRunnable. driverUrl: spark://CoarseGrainedScheduler@192.168.136.135:46344,  executorHostname: big1
17/02/26 23:59:55 INFO ExecutorRunnable: Starting Executor Container
17/02/26 23:59:55 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
17/02/26 23:59:55 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
17/02/26 23:59:55 INFO ExecutorRunnable: Setting up ContainerLaunchContext
17/02/26 23:59:55 INFO ExecutorRunnable: Preparing Local resources
17/02/26 23:59:55 INFO ExecutorRunnable: Prepared Local resources Map(asm-3.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/asm-3.1.jar" } size: 43033 timestamp: 1487901101364 type: FILE visibility: PUBLIC, hive-cli-1.2.1.spark.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/hive-cli-1.2.1.spark.jar" } size: 40812 timestamp: 1487901102736 type: FILE visibility: PUBLIC, commons-lang3-3.3.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-lang3-3.3.2.jar" } size: 412739 timestamp: 1487901102129 type: FILE visibility: PUBLIC, spark-streaming-kafka_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-streaming-kafka_2.10-1.6.1.jar" } size: 284924 timestamp: 1487901112022 type: FILE visibility: PUBLIC, parquet-format-2.3.0-incubating.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-format-2.3.0-incubating.jar" } size: 387188 timestamp: 1487901104900 type: FILE visibility: PUBLIC, tachyon-client-0.8.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/tachyon-client-0.8.2.jar" } size: 2291167 timestamp: 1487901112419 type: FILE visibility: PUBLIC, jackson-module-scala_2.10-2.4.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-module-scala_2.10-2.4.4.jar" } size: 549415 timestamp: 1487901103060 type: FILE visibility: PUBLIC, metrics-core-3.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/metrics-core-3.1.2.jar" } size: 112558 timestamp: 1487901104209 type: FILE visibility: PUBLIC, slf4j-log4j12-1.6.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/slf4j-log4j12-1.6.6.jar" } size: 9711 timestamp: 1487901109149 type: FILE visibility: PUBLIC, pmml-agent-1.1.15.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/pmml-agent-1.1.15.jar" } size: 5278 timestamp: 1487901105265 type: FILE visibility: PUBLIC, spark-mllib_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-mllib_2.10-1.6.1.jar" } size: 4991327 timestamp: 1487901111550 type: FILE visibility: PUBLIC, __spark__.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-yarn_2.10-1.6.1.jar" } size: 561734 timestamp: 1487901112191 type: FILE visibility: PUBLIC, flume-ng-core-1.6.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/flume-ng-core-1.6.0.jar" } size: 370107 timestamp: 1487901102569 type: FILE visibility: PUBLIC, zookeeper-3.4.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/zookeeper-3.4.6.jar" } size: 792964 timestamp: 1487901112691 type: FILE visibility: PUBLIC, oozie-sharelib-oozie-4.3.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/oozie/oozie-sharelib-oozie-4.3.0.jar" } size: 54976 timestamp: 1487901096210 type: FILE visibility: PUBLIC, json4s-jackson_2.10-3.2.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/json4s-jackson_2.10-3.2.10.jar" } size: 39953 timestamp: 1487901103616 type: FILE visibility: PUBLIC, metrics-graphite-3.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/metrics-graphite-3.1.2.jar" } size: 20852 timestamp: 1487901104241 type: FILE visibility: PUBLIC, parquet-common-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-common-1.7.0.jar" } size: 21575 timestamp: 1487901104860 type: FILE visibility: PUBLIC, commons-lang-2.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-lang-2.4.jar" } size: 261809 timestamp: 1487901102081 type: FILE visibility: PUBLIC, avro-mapred-1.7.7-hadoop2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/avro-mapred-1.7.7-hadoop2.jar" } size: 180736 timestamp: 1487901101452 type: FILE visibility: PUBLIC, spark-streaming-flume-sink_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-streaming-flume-sink_2.10-1.6.1.jar" } size: 86038 timestamp: 1487901111931 type: FILE visibility: PUBLIC, spark-launcher_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-launcher_2.10-1.6.1.jar" } size: 65251 timestamp: 1487901111451 type: FILE visibility: PUBLIC, RoaringBitmap-0.5.11.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/RoaringBitmap-0.5.11.jar" } size: 201928 timestamp: 1487901101180 type: FILE visibility: PUBLIC, ivy-2.4.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/ivy-2.4.0.jar" } size: 1282424 timestamp: 1487901102859 type: FILE visibility: PUBLIC, spark-streaming_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-streaming_2.10-1.6.1.jar" } size: 2052495 timestamp: 1487901112129 type: FILE visibility: PUBLIC, kryo-2.22.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/kryo-2.22.jar" } size: 442805 timestamp: 1487901103943 type: FILE visibility: PUBLIC, pyspark.zip -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/hadoop/.sparkStaging/application_1488181659296_0004/pyspark.zip" } size: 283560 timestamp: 1488182389071 type: FILE visibility: PRIVATE, spark-core_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-core_2.10-1.6.1.jar" } size: 11179076 timestamp: 1487901111264 type: FILE visibility: PUBLIC, commons-cli-1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-cli-1.2.jar" } size: 41123 timestamp: 1487901101937 type: FILE visibility: PUBLIC, chill_2.10-0.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/chill_2.10-0.5.0.jar" } size: 221032 timestamp: 1487901101915 type: FILE visibility: PUBLIC, curator-recipes-2.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/curator-recipes-2.5.0.jar" } size: 244782 timestamp: 1487901102457 type: FILE visibility: PUBLIC, parquet-encoding-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-encoding-1.7.0.jar" } size: 285447 timestamp: 1487901104882 type: FILE visibility: PUBLIC, datanucleus-core-3.2.10.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/datanucleus-core-3.2.10.jar" } size: 1890075 timestamp: 1487901102504 type: FILE visibility: PUBLIC, chill-java-0.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/chill-java-0.5.0.jar" } size: 46715 timestamp: 1487901101855 type: FILE visibility: PUBLIC, parquet-hadoop-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-hadoop-1.7.0.jar" } size: 209622 timestamp: 1487901105134 type: FILE visibility: PUBLIC, commons-dbcp-1.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-dbcp-1.4.jar" } size: 160519 timestamp: 1487901102014 type: FILE visibility: PUBLIC, janino-2.7.8.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/janino-2.7.8.jar" } size: 613299 timestamp: 1487901103083 type: FILE visibility: PUBLIC, py4j-0.9-src.zip -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/hadoop/.sparkStaging/application_1488181659296_0004/py4j-0.9-src.zip" } size: 44846 timestamp: 1488182389117 type: FILE visibility: PRIVATE, jackson-mapper-asl-1.9.13.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-mapper-asl-1.9.13.jar" } size: 780664 timestamp: 1487901103036 type: FILE visibility: PUBLIC, log4j-1.2.17.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/log4j-1.2.17.jar" } size: 489884 timestamp: 1487901104102 type: FILE visibility: PUBLIC, py4j.zip -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/py4j.zip" } size: 37562 timestamp: 1487901106810 type: FILE visibility: PUBLIC, jtransforms-2.4.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jtransforms-2.4.0.jar" } size: 764569 timestamp: 1487901103680 type: FILE visibility: PUBLIC, javax.servlet-3.0.0.v201112011016.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/javax.servlet-3.0.0.v201112011016.jar" } size: 200387 timestamp: 1487901103143 type: FILE visibility: PUBLIC, arpack_combined_all-0.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/arpack_combined_all-0.1.jar" } size: 1194003 timestamp: 1487901101335 type: FILE visibility: PUBLIC, jets3t-0.7.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jets3t-0.7.1.jar" } size: 377780 timestamp: 1487901103357 type: FILE visibility: PUBLIC, slf4j-api-1.6.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/slf4j-api-1.6.6.jar" } size: 26176 timestamp: 1487901109117 type: FILE visibility: PUBLIC, breeze_2.10-0.11.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/breeze_2.10-0.11.2.jar" } size: 13689583 timestamp: 1487901101691 type: FILE visibility: PUBLIC, paranamer-2.6.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/paranamer-2.6.jar" } size: 32806 timestamp: 1487901104802 type: FILE visibility: PUBLIC, parquet-hadoop-bundle-1.6.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-hadoop-bundle-1.6.0.jar" } size: 2796935 timestamp: 1487901105193 type: FILE visibility: PUBLIC, pmml-model-1.1.15.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/pmml-model-1.1.15.jar" } size: 656969 timestamp: 1487901105320 type: FILE visibility: PUBLIC, parquet-generator-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-generator-1.7.0.jar" } size: 21243 timestamp: 1487901105087 type: FILE visibility: PUBLIC, py4j-0.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/py4j-0.9.jar" } size: 84451 timestamp: 1487901105427 type: FILE visibility: PUBLIC, akka-actor_2.10-2.3.11.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/akka-actor_2.10-2.3.11.jar" } size: 2658152 timestamp: 1487901101241 type: FILE visibility: PUBLIC, curator-client-2.5.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/curator-client-2.5.0.jar" } size: 68692 timestamp: 1487901102398 type: FILE visibility: PUBLIC, lz4-1.3.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/lz4-1.3.0.jar" } size: 236880 timestamp: 1487901104134 type: FILE visibility: PUBLIC, calcite-core-1.2.0-incubating.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/calcite-core-1.2.0-incubating.jar" } size: 3519262 timestamp: 1487901101798 type: FILE visibility: PUBLIC, spark1.py -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/examples/apps/pythonApp/lib/spark1.py" } size: 747 timestamp: 1488182369956 type: FILE visibility: PUBLIC, xz-1.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/xz-1.0.jar" } size: 94672 timestamp: 1487901112637 type: FILE visibility: PUBLIC, jersey-core-1.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jersey-core-1.9.jar" } size: 458739 timestamp: 1487901103294 type: FILE visibility: PUBLIC, snappy-java-1.1.2.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/snappy-java-1.1.2.jar" } size: 704708 timestamp: 1487901109212 type: FILE visibility: PUBLIC, calcite-linq4j-1.2.0-incubating.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/calcite-linq4j-1.2.0-incubating.jar" } size: 442406 timestamp: 1487901101825 type: FILE visibility: PUBLIC, spark-sql_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-sql_2.10-1.6.1.jar" } size: 4031634 timestamp: 1487901111866 type: FILE visibility: PUBLIC, leveldbjni-all-1.8.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/leveldbjni-all-1.8.jar" } size: 1045744 timestamp: 1487901104014 type: FILE visibility: PUBLIC, jsr305-1.3.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jsr305-1.3.9.jar" } size: 33015 timestamp: 1487901103636 type: FILE visibility: PUBLIC, zkclient-0.3.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/zkclient-0.3.jar" } size: 64009 timestamp: 1487901112662 type: FILE visibility: PUBLIC, oro-2.0.8.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/oro-2.0.8.jar" } size: 65261 timestamp: 1487901104787 type: FILE visibility: PUBLIC, spire_2.10-0.7.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spire_2.10-0.7.4.jar" } size: 7253596 timestamp: 1487901112341 type: FILE visibility: PUBLIC, eigenbase-properties-1.1.5.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/eigenbase-properties-1.1.5.jar" } size: 18482 timestamp: 1487901102525 type: FILE visibility: PUBLIC, commons-codec-1.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/commons-codec-1.4.jar" } size: 58160 timestamp: 1487901101951 type: FILE visibility: PUBLIC, compress-lzf-1.0.3.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/compress-lzf-1.0.3.jar" } size: 79845 timestamp: 1487901102289 type: FILE visibility: PUBLIC, oozie-hadoop-utils-hadoop-2-4.3.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/oozie/oozie-hadoop-utils-hadoop-2-4.3.0.jar" } size: 11851 timestamp: 1487901096191 type: FILE visibility: PUBLIC, opencsv-2.3.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/opencsv-2.3.jar" } size: 19827 timestamp: 1487901104675 type: FILE visibility: PUBLIC, netty-all-4.0.29.Final.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/netty-all-4.0.29.Final.jar" } size: 2054931 timestamp: 1487901104589 type: FILE visibility: PUBLIC, jetty-util-6.1.26.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jetty-util-6.1.26.jar" } size: 177131 timestamp: 1487901103412 type: FILE visibility: PUBLIC, parquet-jackson-1.7.0.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/parquet-jackson-1.7.0.jar" } size: 1048110 timestamp: 1487901105228 type: FILE visibility: PUBLIC, kafka-clients-0.8.2.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/kafka-clients-0.8.2.1.jar" } size: 324010 timestamp: 1487901103773 type: FILE visibility: PUBLIC, jackson-annotations-2.4.4.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jackson-annotations-2.4.4.jar" } size: 38597 timestamp: 1487901102878 type: FILE visibility: PUBLIC, akka-slf4j_2.10-2.3.11.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/akka-slf4j_2.10-2.3.11.jar" } size: 15525 timestamp: 1487901101307 type: FILE visibility: PUBLIC, spark-network-shuffle_2.10-1.6.1.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/spark-network-shuffle_2.10-1.6.1.jar" } size: 54995 timestamp: 1487901111636 type: FILE visibility: PUBLIC, jersey-server-1.9.jar -> resource { scheme: "hdfs" host: "bigmaster" port: 9000 file: "/user/oozie/share/lib/spark/jersey-server-1.9.jar" } size: 713089 timestamp: 1487901103320 type: FILE visibility: PUBLIC, spark-graphx_2.10-1.6.1.jar -> resource { scheme: "hdfs" host
最低0.47元/天 解锁文章
flash胜龙
关注
  • 0
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
傅立叶变换java代码
01-07
傅立叶变换 java gui界面 可实现一维二维傅立叶变换和一维二维傅立叶反变换。
yarn错题本
epitomizelu的专栏
10-10 860
提交到yarn上的job显示running状态，但是executor的日志持续打印如下告警： WARN YarnScheduler:66 - Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 表面上看是集群资源不足，但从yarn资源管理界面看cup和内存非常宽裕，单个结点的资源也是足够的，
Spark WARN cluster.ClusterScheduler: Initial job has not accepted any resources;check your cluster
热门推荐
caimo的专栏
04-13 2万+
WARN cluster.ClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory spark cluster
WARN TaskSchedulerImpl: Initial job has not accepted any resources； check your……解决方法，两种可能原因
weixin_48620718的博客
04-14 8594
WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources，意思是：初始作业未接受任何资源；请检查群集UI以确保工作进程已注册并且有足够的资源。 出错可能有两种原因。 第一种：是spark节点的内存满了，内存可以在spark的配置文件—>spark-
Initial job has not accepted any resources； check your cluster UI to ensure...
succeedloveaaaa的专栏
03-21 3443
Spark on YARN执行任务driver端报错： org.apache.spark.scheduler.cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 查了下网上出现这个问题的原因挺多的，下面记录下我遇到的情况： 1、先讲下基本配置
解决Python出现_warn_unsafe_extraction问题的方法
09-21
总的来说，解决“_warn_unsafe_extraction”问题的关键在于理解Python生态系统中不同库之间的依赖关系，以及如何通过调整这些依赖来解决不兼容性问题。在遇到这类问题时，检查错误日志，确定涉及的库和它们的版本，...
购物车.zip_jsp_warn2cf_购物车
07-14
一个比较简单的web页面的购物车参考程序
node.js中的console.warn方法使用说明
10-25
主要介绍了node.js中的console.warn方法使用说明,本文介绍了console.warn的方法说明、语法、接收参数、使用实例和实现源码,需要的朋友可以参考下
[NPM解决方案]npm WARN deprecated core-js2.6.11,core-js小于3 is no longer maintained
01-07
【NPM解决方案】npm WARN deprecated core-js@2.6.11: core-js@< 3 is no longer maintained and not recommended for usage Administrator@GuoXinYu MINGW64 /d $ npm install -g vuepress npm WARN deprecated ...
spark on yarn，Initial job has not accepted any resources
cs_joel的博客
02-15 2470
WARN cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 执行spark作业在yarn上时，一直提示以上信息，通过yarn RM
Spark提交Jar任务警告: Initial job has not accepted any resources;
时间静止
12-20 2270
错误信息描述 在Spark提交任务时, 会一直出现下面警告 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
解决报错： YarnScheduler: Initial job has not accepted any resources
发现问题，并解决问题，批判性思维
08-02 1429
解决报错： YarnScheduler: Initial job has not accepted any resources。
spark报错：WARN TaskSchedulerImpl: Initial job has not accepted any resources； check your cluster UI...
Iv_zzy的博客
11-05 7335
Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Spark Core：第二章 spark集群遇到的问题解决
落落free的博客
05-02 890
spark集群遇到的问题解决 文章目录spark集群遇到的问题解决Q1：Standalone模式提交任务Q2：YARN模式提交任务Q3：Yarn application has already ended!Q4：执行Yarn模式Q5：webui端的alive worker节点显示为0 Q1：Standalone模式提交任务 错误描述： 19/05/01 05:34:01 ERROR Spa...
测试集群，任务运行失败
weixin_42319892的博客
10-20 341
动态任务失败
Spark错误问题记录
再难也要坚持的博客
03-11 1704
Spark错误记录java.lang.IllegalArgumentException: Required executor memory (1024), overhead (384 MB), and PySpark memory (0 MB) is above the max threshold (1024 MB) of this cluster! Please check the values...
Spark源码——Spark on YARN Container资源申请分配、Executor的启动
aof
03-07 1783
文章目录Container资源申请分配Executor的启动 Container资源申请分配 Executor的启动
WARN&nbsp;YarnClusterScheduler:&nbsp;not&nbsp;a…
flash胜龙的博客
04-21 703
解决了pyspark找不到"python"的问题以后： 【新问题】 查看目标节点8042端口web ui中application list里面的container的logs，发现无法分配资源的老问题： 17/02/26 22:33:11 WARN YarnClusterScheduler: Initial job has not accepted any resources; check your
initial job has not accepted any resources的spark错误解决办法
大数据挖掘SparkExpert的博客
03-04 9739
在运行多个spark应用程序的时候，经常会出现initial job has not accepted any resources的错误。 而如果用spark-submit方式提交的话，经常是结果不显示。而在监控网页上看到的是状态一直在waiing.如下所示。 经过仔细检查，发现是CORES的问题，由于之前的程序将CORES全部占用，而后来的任务没有足够的CPU来支撑，
如何解决health HEALTH_WARN
最新发布
06-13
"health HEALTH_WARN"是Elasticsearch中的一个警告，它意味着集群的健康状况被评为黄色。这通常是由于某些节点不可用或磁盘空间不足等因素引起的。 要解决此问题，需要找出导致黄色健康状态的根本原因。可以通过运行以下命令来检查集群的健康状况和节点状态： ``` curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' curl -XGET 'http://localhost:9200/_cat/nodes?v' ``` 如果节点不可用，则需要诊断故障并修复节点。如果磁盘空间不足，则需要清理磁盘或添加更多磁盘空间。 一旦解决了根本原因，集群的健康状态将自动更新为绿色。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值