【spark】due to AM Container for x exited with exitCode: -1000.which is below the minimum allowed 1000

于 2023-05-03 21:26:50 发布
阅读量1k 收藏
点赞数
分类专栏： 大数据问题专栏 文章标签： spark yarn hadoop
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/afivestudy/article/details/130477107
版权
在开发一个Spark处理Kudu的程序时，通过spark-submit提交到YARN集群出现错误，AM Container退出码为-1000，低于最小允许的1000。解决方法是检查并修改YARN配置中的`min.user.id`参数，将其值设置为500，然后重启YARN服务，最终成功运行Spark程序并完成Kudu操作。
摘要由CSDN通过智能技术生成

        （上方报错由于字数限制，显示不全，具体报错搜索下面）

        这次，在实际开发测试过程中，开发一个spark操作kudu的程序。打包完成后，上传到集群上，使用spark-submit提交程序，发生报错。

        提交程序的命令如下：

 spark-submit \
 --master yarn \
 --deploy-mode cluster \
 --queue spark \
 --driver-cores 1 \
 --driver-memory 1G \
 --executor-cores 1 \
 --executor-memory 1G \
 --num-executors 1 \
 --class kudu.spark.SparkKuduTest \
 Kudu-1.0-SNAPSHOT-jar-with-dependencies.jar

        运行报错如下：

[root@cdh01 ~]# spark-submit --master yarn --deploy-mode client --queue spark --driver-cores 1 --driver-memory 1G --executor-cores 1 --executor-memory 1G --num-executors 1 --class kudu.spark.SparkKuduTest Kudu-1.0-SNAPSHOT-jar-with-dependencies.jar
23/05/03 20:30:33 INFO security.UserGroupInformation: Login successful for user spark/spark@HADOOP.COM using keytab file /etc/security/keytab/spark.keytab. Keytab auto renewal enabled : false
23/05/03 20:30:33 INFO spark.SparkContext: Running Spark version 2.4.0-cdh6.3.2
23/05/03 20:30:33 INFO logging.DriverLogger: Added a local log appender at: /tmp/spark-6d3361b1-8525-4274-bca3-88c0d8e70dac/__driver_logs__/driver.log
23/05/03 20:30:33 INFO spark.SparkContext: Submitted application: SparkKudu
23/05/03 20:30:33 INFO spark.SecurityManager: Changing view acls to: root,spark
23/05/03 20:30:33 INFO spark.SecurityManager: Changing modify acls to: root,spark
23/05/03 20:30:33 INFO spark.SecurityManager: Changing view acls groups to: 
23/05/03 20:30:33 INFO spark.SecurityManager: Changing modify acls groups to: 
23/05/03 20:30:33 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root, spark); groups with view permissions: Set(); users  with modify permissions: Set(root, spark); groups with modify permissions: Set()
23/05/03 20:30:33 INFO util.Utils: Successfully started service 'sparkDriver' on port 45095.
23/05/03 20:30:33 INFO spark.SparkEnv: Registering MapOutputTracker
23/05/03 20:30:33 INFO spark.SparkEnv: Registering BlockManagerMaster
23/05/03 20:30:33 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
23/05/03 20:30:33 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
23/05/03 20:30:33 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-59fc6e85-710a-49d9-b50f-55959dc5b217
23/05/03 20:30:33 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
23/05/03 20:30:33 INFO spark.SparkEnv: Registering OutputCommitCoordinator
23/05/03 20:30:33 INFO util.log: Logging initialized @1686ms
23/05/03 20:30:33 INFO server.Server: jetty-9.3.z-SNAPSHOT, build timestamp: 2018-09-05T05:11:46+08:00, git hash: 3ce520221d0240229c862b122d2b06c12a625732
23/05/03 20:30:33 INFO server.Server: Started @1747ms
23/05/03 20:30:33 INFO server.AbstractConnector: Started ServerConnector@232024b9{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
23/05/03 20:30:33 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6aa648b9{/jobs,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2f6bbeb0{/jobs/json,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1b1637e1{/jobs/job,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@64711bf2{/jobs/job/json,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@169da7f2{/stages,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3c1e23ff{/stages/json,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@ceb4bd2{/stages/stage,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@56bc3fac{/stages/stage/json,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@df4b72{/stages/pool,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2ba45490{/stages/pool/json,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@37ff4054{/storage,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@894858{/storage/json,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7af707e0{/storage/rdd,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@737edcfa{/storage/rdd/json,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3ecedf21{/environment,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@74cf8b28{/environment/json,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@36c54a56{/executors,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3359c978{/executors/json,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7ba63fe5{/executors/threadDump,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@73386d72{/executors/threadDump/json,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6f330eb9{/static,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@79c5636f{/,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1929425f{/api,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2617f816{/jobs/job/kill,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@676f0a60{/stages/stage/kill,null,AVAILABLE,@Spark}
23/05/03 20:30:33 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://cdh01.shokod.com:4040
23/05/03 20:30:33 INFO spark.SparkContext: Added JAR file:/root/Kudu-1.0-SNAPSHOT-jar-with-dependencies.jar at spark://cdh01.shokod.com:45095/jars/Kudu-1.0-SNAPSHOT-jar-with-dependencies.jar with timestamp 1683117033813
23/05/03 20:30:33 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
23/05/03 20:30:33 INFO conf.HiveConf: Found configuration file file:/etc/hive/conf.cloudera.hive/hive-site.xml
23/05/03 20:30:33 INFO security.YARNHadoopDelegationTokenManager: Attempting to load user's ticket cache.
23/05/03 20:30:35 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
23/05/03 20:30:35 INFO security.HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-144794697_1, ugi=spark/spark@HADOOP.COM (auth:KERBEROS)]] with renewer yarn/cdh02.shokod.com@HADOOP.COM
23/05/03 20:30:35 INFO hdfs.DFSClient: Created token for spark: HDFS_DELEGATION_TOKEN owner=spark/spark@HADOOP.COM, renewer=yarn, realUser=, issueDate=1683117035838, maxDate=1683721835838, sequenceNumber=21, masterKeyId=46 on ha-hdfs:nameserviceHA
23/05/03 20:30:35 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
23/05/03 20:30:35 INFO security.HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-144794697_1, ugi=spark/spark@HADOOP.COM (auth:KERBEROS)]] with renewer spark/spark@HADOOP.COM
23/05/03 20:30:35 INFO hdfs.DFSClient: Created token for spark: HDFS_DELEGATION_TOKEN owner=spark/spark@HADOOP.COM, renewer=spark, realUser=, issueDate=1683117035867, maxDate=1683721835867, sequenceNumber=22, masterKeyId=46 on ha-hdfs:nameserviceHA
23/05/03 20:30:35 INFO security.HadoopFSDelegationTokenProvider: Renewal interval is 86400035 for token HDFS_DELEGATION_TOKEN
23/05/03 20:30:36 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
23/05/03 20:30:36 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm43
23/05/03 20:30:36 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
23/05/03 20:30:36 INFO conf.Configuration: resource-types.xml not found
23/05/03 20:30:36 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
23/05/03 20:30:36 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (4096 MB per container)
23/05/03 20:30:36 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
23/05/03 20:30:36 INFO yarn.Client: Setting up container launch context for our AM
23/05/03 20:30:36 INFO yarn.Client: Setting up the launch environment for our AM container
23/05/03 20:30:36 INFO yarn.Client: Preparing resources for our AM container
23/05/03 20:30:36 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
23/05/03 20:30:36 INFO yarn.Client: Uploading resource file:/tmp/spark-6d3361b1-8525-4274-bca3-88c0d8e70dac/__spark_conf__6944142613266959994.zip -> hdfs://nameserviceHA/user/spark/.sparkStaging/application_1683105646105_0007/__spark_conf__.zip
23/05/03 20:30:36 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
23/05/03 20:30:36 INFO spark.SecurityManager: Changing view acls to: root,spark
23/05/03 20:30:36 INFO spark.SecurityManager: Changing modify acls to: root,spark
23/05/03 20:30:36 INFO spark.SecurityManager: Changing view acls groups to: 
23/05/03 20:30:36 INFO spark.SecurityManager: Changing modify acls groups to: 
23/05/03 20:30:36 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root, spark); groups with view permissions: Set(); users  with modify permissions: Set(root, spark); groups with modify permissions: Set()
23/05/03 20:30:36 INFO yarn.Client: Submitting application application_1683105646105_0007 to ResourceManager
23/05/03 20:30:36 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
23/05/03 20:30:37 INFO impl.YarnClientImpl: Submitted application application_1683105646105_0007
23/05/03 20:30:37 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
23/05/03 20:30:38 INFO yarn.Client: Application report for application_1683105646105_0007 (state: ACCEPTED)
23/05/03 20:30:38 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: [Wed May 03 20:30:37 +0800 2023] Application is Activated, waiting for resources to be assigned for AM.  Details : AM Partition = <DEFAULT_PARTITION> ; Partition Resource = <memory:9216, vCores:12> ; Queue's Absolute capacity = 40.0 % ; Queue's Absolute used capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; 
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: spark
	 start time: 1683117036782
	 final status: UNDEFINED
	 tracking URL: http://cdh03.shokod.com:8088/proxy/application_1683105646105_0007/
	 user: spark
23/05/03 20:30:38 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
23/05/03 20:30:39 INFO yarn.Client: Application report for application_1683105646105_0007 (state: FAILED)
23/05/03 20:30:39 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: Application application_1683105646105_0007 failed 2 times due to AM Container for appattempt_1683105646105_0007_000002 exited with  exitCode: -1000
Failing this attempt.Diagnostics: [2023-05-03 20:30:38.488]Application application_1683105646105_0007 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is spark
main : requested yarn user is spark
Requested user spark is not whitelisted and has id 978,which is below the minimum allowed 1000

For more detailed output, check the application tracking page: http://cdh03.shokod.com:8088/cluster/app/application_1683105646105_0007 Then click on links to logs of each attempt.
. Failing the application.
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: spark
	 start time: 1683117036782
	 final status: FAILED
	 tracking URL: http://cdh03.shokod.com:8088/cluster/app/application_1683105646105_0007
	 user: spark
23/05/03 20:30:39 INFO yarn.Client: Deleted staging directory hdfs://nameserviceHA/user/spark/.sparkStaging/application_1683105646105_0007
23/05/03 20:30:39 ERROR cluster.YarnClientSchedulerBackend: The YARN application has already ended! It might have been killed or the Application Master may have failed to start. Check the YARN application logs for more details.
23/05/03 20:30:39 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Application application_1683105646105_0007 failed 2 times due to AM Container for appattempt_1683105646105_0007_000002 exited with  exitCode: -1000
Failing this attempt.Diagnostics: [2023-05-03 20:30:38.488]Application application_1683105646105_0007 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is spark
main : requested yarn user is spark
Requested user spark is not whitelisted and has id 978,which is below the minimum allowed 1000

For more detailed output, check the application tracking page: http://cdh03.shokod.com:8088/cluster/app/application_1683105646105_0007 Then click on links to logs of each attempt.
. Failing the application.
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:95)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:186)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:511)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
	at kudu.spark.SparkKuduTest$.createTable(SparkKuduTest.scala:41)
	at kudu.spark.SparkKuduTest$.main(SparkKuduTest.scala:20)
	at kudu.spark.SparkKuduTest.main(SparkKuduTest.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/05/03 20:30:39 INFO server.AbstractConnector: Stopped Spark@232024b9{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
23/05/03 20:30:39 INFO ui.SparkUI: Stopped Spark web UI at http://cdh01.shokod.com:4040
23/05/03 20:30:39 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
23/05/03 20:30:39 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
23/05/03 20:30:39 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
23/05/03 20:30:39 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
23/05/03 20:30:39 INFO cluster.YarnClientSchedulerBackend: Stopped
23/05/03 20:30:39 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/05/03 20:30:39 INFO memory.MemoryStore: MemoryStore cleared
23/05/03 20:30:39 INFO storage.BlockManager: BlockManager stopped
23/05/03 20:30:39 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
23/05/03 20:30:39 WARN metrics.MetricsSystem: Stopping a MetricsSystem that is not running
23/05/03 20:30:39 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/05/03 20:30:45 INFO spark.SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: Application application_1683105646105_0007 failed 2 times due to AM Container for appattempt_1683105646105_0007_000002 exited with  exitCode: -1000
Failing this attempt.Diagnostics: [2023-05-03 20:30:38.488]Application application_1683105646105_0007 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is spark
main : requested yarn user is spark
Requested user spark is not whitelisted and has id 978,which is below the minimum allowed 1000

For more detailed output, check the application tracking page: http://cdh03.shokod.com:8088/cluster/app/application_1683105646105_0007 Then click on links to logs of each attempt.
. Failing the application.
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:95)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:186)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:511)
	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
	at kudu.spark.SparkKuduTest$.createTable(SparkKuduTest.scala:41)
	at kudu.spark.SparkKuduTest$.main(SparkKuduTest.scala:20)
	at kudu.spark.SparkKuduTest.main(SparkKuduTest.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/05/03 20:30:45 INFO util.ShutdownHookManager: Shutdown hook called
23/05/03 20:30:45 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-6d3361b1-8525-4274-bca3-88c0d8e70dac
23/05/03 20:30:45 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-b12e58dd-27a8-4a2c-9958-b39f9044baad

        查看具体的报错日志：

Application application_1683105646105_0007 failed 2 times due to AM Container for appattempt_1683105646105_0007_000002 exited with exitCode: -1000

Failing this attempt.Diagnostics: [2023-05-03 20:30:38.488]Application application_1683105646105_0007 initialization failed (exitCode=255) with output: main : command provided 0

main : run as user is spark

main : requested yarn user is spark

Requested user spark is not whitelisted and has id 978,which is below the minimum allowed 1000

For more detailed output, check the application tracking page: http://cdh03.shokod.com:8088/cluster/app/application_1683105646105_0007 Then click on links to logs of each attempt.

. Failing the application.

        从上面可以看到，用户的id是低于最低1000限制。 所以，我猜测是yarn集群的配置出现了问题，在YARN的配置中查找关于“id”的相关配置。找到最小用户ID“min.user.id”（默认是1000）比较符合，修改阈值为500。重启YARN服务

 

        集群上重新运行程序，运行成功：

        自身也在kudu上验证没有问题！问题完美解决了！

 

大数据解说猿SHOKO
关注
专栏目录
Yarn初始化异常：Not able to initialize user directories.exitCode:-1000
superMario
01-30 2993
最近碰到这么一个异常： Application application_1528180019666_0366 failed 2 times due to AM Container for appattempt_1528180019666_0366 exited with exitCode: -1000 For more detailed output, check application tra...
spark on yarn exitCode: -104
angela_2016的博客
05-20 1840
执行spark任务时，每次启动后，少则一个小时，多则两三天左右，任务就会死掉，yarn日志报错见下图： AM Container for appattempt_1554609747730_49028_000001 exited with exitCode: -104 For more detailed output, check application tra...
spark on yarn：due to AM Container for appattempt_xx_xx exited with exitCode:-1000
weixin_44455388的博客
04-28 3503
新部署的hadoop和spark集群，使用spark on yarn模式执行一个简单的spark程序包，spark程序的代码逻辑是读取oracle数据库，读取后对数据做简单处理，然后保存。比较匪夷所思的是，spark代码逻辑执行成功，经过处理的数据成功保存到数据库，但是yarn页面显示spark程序报错。 报错信息如下： 按这篇文章（https://www.e-learn.cn/content/wangluowenzhang/874518）的解决方法，无法解决。 网上找了很多都没有找到解决方。 由于是新部
解决 Application xxx failed 2 times due to AM Container for xxx exited with exitCode: 13 问题
qq_37354060的博客
03-21 8638
出现问题要首先检查hostname是不是与hadoop、spark配置的一致，不要想当然的认为是一样的，先检查一下最基础的，如果还是不能解决，就要在网上搜索一下了，通常情况下，问题产生的原因都是因为忽略了某个地方的小细节。
spark yarn 模式提交和sqoop从mysql导入到hdfs，hive均报错exitCode:-1000
weixin_47340211的博客
04-24 258
但千万注意格式化之前要清空data和logs文件下内容。最后试了一下重新格式化Namenode。然后去yarn界面查看历史。首先看下统一的报错内容。rm -rf 文件夹\。
YARN调度报错: exitCode: -1000 due to: File does not exist: hdfs
qq_41669665的博客
10-17 1801
把setMaster（"local"）改为setMaster("yarn-cluster")
Application application_xxx failed 2 times due to AM Container for attempt_xxx exited withexitCode:
m0_53867640的博客
08-01 4340
通过上面的两张图片不难看出，主要是存在两个问题。结果显示块状态是健康的，即并没有块损坏这一说法。
Process exited with an error: -10000(Exit value: -10000)
Arvan91的博客
08-11 1824
打包出现如标题所示异常即Process exited with an error: -10000(Exit value: -10000)一开始一直拿警告信息到网上查找解决方案，但是有时候可以打包成功有时候又不可以，一开始也是这样。根据该博客指引把对应的虚拟机内存空间调大即可解决当前error。如果还是失败，就一直往上加下面的参数值。 java.lang.StackOverflowError...
Container exited with a non-zero exit code 1. Error file: prelaunch.err.org.apache.hadoop.mapreduce.
colby_chenlun的博客
05-31 2万+
这两天闲的慌，装了一下新版本的HADOOP 3.2.1，的HA，装好之后，测试wordcount程序时报下面的错误 For more detailed output, check the application tracking page: http://hadoop102:8088/cluster/app/application_1590917926429_0002 Then click on links to logs of each attempt. . Failing the applicati
Spark on Yarn的常见错误
muyingmiao的专栏
10-25 635
1. 将spark作业提交到yarn上执行 spark仅仅作为一个客户端 ./spark-submit \ --class org.apache.spark.examples.SparkPi \ --master yarn \ /home/hadoop/app/spark-2.4.4-bin-2.6.0-cdh5.15.1/examples/jars/spark-examples_2.11-...
Application finished with exit code: 1
guanqifighting的博客
02-02 1569
Maven启动SpringBoot，报错：Application finished with exit code: 1
开启 Kerberos 安全的大数据环境中，Yarn Container 启动失败导致作业失败
明哥的IT随笔
09-16 1914
大数据问题排查系列 - 开启 Kerberos 安全的大数据环境中，Yarn Container 启动失败导致 spark/hive 作业失败 前言 大家好，我是明哥！ 最近在若干个不同客户现场，都遇到了 大数据集群中开启 Kerberos 后，spark/hive 作业提交到YARN 后，因 YARN Container 启动失败作业无法执行的情况，在此总结下背后的知识点，跟大家分享下，希望大家有所收获。 1 问题1问题现象 某客户现场，大数据集群中开启了 kerberos 安全认证，提交 hive on
TezSession has already shutdown.Application XXX failed 2 times due to AM Container
weixin_43923463的博客
03-26 2089
错误信息 org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1648284477641_0003 failed 2 times due to AM Container for appattempt_1648284477641_0003_000002 exited with exitCode: 1 报错原因 hadoop集群上从机上运行的Container试
Flink 启动 yarn-session.sh
qq_40471761的博客
07-20 4630
1.内存不足 启动命令 bin/yarn-session.sh -n 2 -s 2 -jm 1024 -tm 1024 解决办法：将 hadoop 的检查虚拟内存关闭掉 在yarn-site.xml 中添加 <property> <name>yarn.nodemanager.vmem-check-enabled</name> ...
‘conda‘ 不是内部或外部命令，也不是可运行的程序 或批处理文件。
阿生
05-02 5万+
'conda' 不是内部或外部命令，也不是可运行的程序 或批处理文件。 今天安装Anaconda，安装之后运行发现总是提示命令找不到，但是我检查环境变量发现环境变量已经配置好了，而且看了很多次发现没有错误 原来是因为我用了ListArr软件打开的cmd窗口，这款软件打开的cmd窗口没有更新环境变量，依旧是开机之前的配置 解决办法： 在windows左下角自带的搜索工具的（或wi...
Name node is in safe mode.（NameNode处于安全模式）
阿生
03-29 1万+
当我给自己的程序打包，然后放入Hadoop中运行时出现异常 17/03/29 22:38:37 INFO client.RMProxy: Connecting to ResourceManager at master/172.18.63.28:8032 17/03/29 22:38:37 WARN mapreduce.JobResourceUploader: Hadoop command-lin
升级CDH集群的JDK
阿生
09-05 559
升级CDH集群的JDK 更改所有节点的java命令 [root@a01 ~]# whereis java java: /usr/bin/java /etc/java /usr/lib/java /usr/share/java /usr/share/man/man1/java.1.gz [root@a01 ~]# ll /usr/bin/java lrwxrwxrwx. 1 root root 22...
程序退出码 exit code 分析
wenyixicodedog的博客
08-11 8886
当进程因收到信号被终止执行退出后，父进程可以通过调用wait或waitpid得到它的exit code。 我们经常会见到类似如下程序退出信息，但是一直却不知道他们各自代表什么含义。 程序退出示例 程序进程被各信号终止的退出状态码代表含义总结如下： 进一步总结如下： 能使进程被终止执行并产生core dump的信号，它的退出状态码是信号编号+128，比如SI...
Linux进程被信号杀死后退出状态码(exit code)的分析
热门推荐
halfclear的博客
05-27 6万+
linux系统下，进程对信号的默认响应方式有5种： 1）  忽略信号，即当做没收到信号一样； 2）  终止进程； 3）  产生核心转储文件，同时进程终止； 4）  停止进程，即暂停进程的执行； 5）  于之前的暂停之后恢复执行； 处理方式2和3都会导致进程终止执行立即退出，处理方式1、4、5不会造成进程终止执行。因此，可以将信号粗略的分为两类，一类是会导致进程终止并退出的信号；另一类则
2023-06-09 20:54:26 ERROR Client:70 - Application diagnostics message: Application application_1686294905941_0017 failed 2 times due to AM Container for appattempt_1686294905941_0017_000002 exited with exitCode: 13 Failing this attempt.Diagnostics: Exception from container-launch. Container id: container_1686294905941_0017_02_000001 Exit code: 13
最新发布
06-10
根据你的问题，我猜测你遇到了一个关于 Hadoop 应用程序运行失败的问题。根据错误信息，AM 容器启动失败，可能是由于一些异常导致的。具体的原因需要根据日志信息进行分析，可能是程序代码出现了异常或配置不正确导致的。建议你仔细查看日志信息，以便找到具体的原因并解决问题。如果你无法解决，可以提供更多详细信息并寻求帮助。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值