flink on yarn模式出现The main method caused an error: Could not deploy Yarn job cluster问题排查+解决

最新推荐文章于 2024-03-16 14:12:10 发布
最新推荐文章于 2024-03-16 14:12:10 发布
阅读量2.5w 收藏 9
点赞数 6
分类专栏： Flink
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/appleyuchi/article/details/106627117
版权

报错复现:

flink run -m yarn-cluster -p 2 -yjm 700m -ytm 1024m -c WordCount target/bbb-1.0-SNAPSHOT.jar

完整报错如下:

 The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Could not deploy Yarn job cluster.
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:662)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
	at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:893)
	at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:966)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:966)
Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster.
	at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:398)
	at org.apache.flink.client.deployment.executors.AbstractJobClusterExecutor.execute(AbstractJobClusterExecutor.java:70)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1733)
	at org.apache.flink.streaming.api.environment.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:94)
	at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:63)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1620)
	at WordCount.main(WordCount.java:47)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
	... 11 more
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment. 
Diagnostics from YARN: Application application_1591614969089_0002 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1591614969089_0002_000001 exited with  exitCode: 1
Failing this attempt.Diagnostics: [2020-06-08 19:18:12.457]Exception from container-launch.
Container id: container_1591614969089_0002_01_000001
Exit code: 1

[2020-06-08 19:18:12.466]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :

[2020-06-08 19:18:12.467]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :

For more detailed output, check the application tracking page: http://Desktop:8188/applicationhistory/app/application_1591614969089_0002 Then click on links to logs of each attempt.
. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1591614969089_0002
	at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:999)
	at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:488)
	at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:391)
	... 22 more
2020-06-08 19:18:12,659 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Cancelling deployment from Deployment Failure Hook
2020-06-08 19:18:12,660 INFO  org.apache.hadoop.yarn.client.RMProxy                         - Connecting to ResourceManager at Desktop/192.168.0.103:8032
2020-06-08 19:18:12,661 INFO  org.apache.hadoop.yarn.client.AHSProxy                        - Connecting to Application History server at Desktop/192.168.0.103:10201
2020-06-08 19:18:12,661 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Killing YARN application
2020-06-08 19:18:12,668 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Killed application application_1591614969089_0002
2020-06-08 19:18:12,769 INFO  org.apache.flink.yarn.YarnClusterDescriptor                   - Deleting files in hdfs://Desktop:9000/user/appleyuchi/.flink/application_1591614969089_0002.

比较难排查的一个报错，注意确保HADOOP的日志服务器打开，即确保jps中有：

JobHistoryServer,启动命令为:

"$HADOOP_HOME/bin/mapred --daemon start historyserver"

打开时间线服务器

yarn timelineserver

进行完上述操作后，yarn界面的各个端口应该都能打开了。
#######################################################################################

然后在yarn界面的log中看到如下报错:

2020-06-08 19:21:02,071 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Shutting YarnJobClusterEntrypoint down with application status FAILED. Diagnostics org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent.
	at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:261)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:215)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:518)
	at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:119)
Caused by: java.net.BindException: Could not start rest endpoint on any port in port range 8082
	at org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:228)
	at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:165)
	... 9 more
.
2020-06-08 19:21:02,076 INFO  org.apache.flink.runtime.blob.BlobServer                      - Stopped BLOB server at 0.0.0.0:37633
2020-06-08 19:21:02,077 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Stopping Akka RPC service.
2020-06-08 19:21:02,082 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Stopping Akka RPC service.
2020-06-08 19:21:02,087 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Shutting down remote daemon.
2020-06-08 19:21:02,088 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remote daemon shut down; proceeding with flushing remote transports.
2020-06-08 19:21:02,095 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Shutting down remote daemon.
2020-06-08 19:21:02,095 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remote daemon shut down; proceeding with flushing remote transports.
2020-06-08 19:21:02,110 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remoting shut down.
2020-06-08 19:21:02,110 INFO  akka.remote.RemoteActorRefProvider$RemotingTerminator         - Remoting shut down.
2020-06-08 19:21:02,130 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Stopped Akka RPC service.
2020-06-08 19:21:02,131 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService              - Stopped Akka RPC service.
2020-06-08 19:21:02,132 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint         - Could not start cluster entrypoint YarnJobClusterEntrypoint.
org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint YarnJobClusterEntrypoint.
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:187)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:518)
	at org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:119)
Caused by: org.apache.flink.util.FlinkException: Could not create the DispatcherResourceManagerComponent.
	at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:261)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:215)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168)
	... 2 more
Caused by: java.net.BindException: Could not start rest endpoint on any port in port range 8082
	at org.apache.flink.runtime.rest.RestServerEndpoint.start(RestServerEndpoint.java:228)
	at org.apache.flink.runtime.entrypoint.component.DefaultDispatcherResourceManagerComponentFactory.create(DefaultDispatcherResourceManagerComponentFactory.java:165)
	... 9 more

##############################################################

端口问题，但是这个端口并没有占用啊，所以我也懵逼了一会儿。

犯错原因：

这两个文件中的端口要保持统一，我忘记修改masters文件了，从而导致了上述复杂的报错。

这里之所以默认的8081要改成8082是因为8081被spark给占用了，所以我当时修改完flink-conf.yaml就忘乎所以了。

 

最终解决方案:

flink-conf.yaml:rest.port: 8082

masters:Desktop:8082

然后别忘记这两个文件同步更新到集群中的其他节点。

关闭眼前的所有终端，重新开一个终端，因为配置文件只有在你开启新终端的情况下才会生效。


 

 

 

 

 

Flink : exitCode=1 the main method caused an error: could not deploy yarn job cluster
九师兄
07-03 3868
1.美图 2.背景 执行一个flink on yarn命令 su -hdfs -c "export HADOOP_CONF_DIR=xxx && export HADOOP_CLASSPATH=`hadoop classpath ` && /soft/flink/bin/flink run -m yarn-cluster -ynm xxx -yn 3 -yjm ...
Flink任务执行错误：无法部署YARN集群大数据任务
TechWhiz的博客
08-17 966
总结一下，当我们尝试将Flink任务部署到YARN集群时，有时会遇到"exitCode= the main method caused an error: could not deploy yarn job cluster"的错误。通过确保YARN集群正常运行、检查Flink和YARN的版本兼容性、正确配置Flink和YARN的相关配置文件，以及重新启动相关服务，我们可以解决这个问题。确保YARN集群正常运行：在提交Flink任务之前，确保YARN集群已经正确安装和配置，并且所有的节点都处于正常运行状态。
The main method caused an error: Could not deploy Yarn job cluster.
qq_34009542的博客
12-22 3228
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Could not deploy Yarn job cluster. Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster. Caused by...
flink on yarn模式The main method caused an error: Could not deploy Yarn job cluster
RUIMENG061511332的博客
07-06 3004
Flink提交任务到yarn集群报错，The main method caused an error: Could not deploy Yarn job cluster 本人的原因是： flink版本不一致造成的，本地开发环境使用了flink1.8.0, 提交到yarn集群，是通过flink1.10.2安装包下的/bin/flink脚本提交的任务。。 ...
Flink 任务提交失败 Could not deploy Yarn job cluster
Baron_ND的博客
09-08 1万+
1.Flink提价任务，读取Hbase数据和hdfs数据，需要单独倒入一些hbase的依赖jar，所以加入了maven依赖，本地能正常测试运行，但是发布到yarn上运行报错，所以测试读取hdfs文件的时候，去掉依赖就可以了。 2.加入打包依赖的时候，注意加入过滤不需要的文件，不然会报错。 org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Could not depl.
任务提交时，报错： Could not deploy Yarn job cluster
东城庞太师
07-26 7526
如图：任务提交时，报错：Could not deploy Yarn job cluster 原因：我们往下看 原因：设置的内存超过了限制。 解决：修改内存大小设置 yarn.scheduler.maximum-allocation-mb yarn.nodemanager.resource.memory-mb ...
flink部署yarn错误：Couldn‘t deploy Yarn session cluster
饶一熊的博客
04-09 8575
Couldn’t deploy Yarn session cluster 错误信息 flink on yarn启动yarn-session时, 出现yarn-session无法部署错误 并且提示2.1G的虚拟内存使用了2.2GB内存. ... org.apache.flink.client.deployment.ClusterDeploymentException: `Couldn't deploy Yarn session cluster` at org.apache.flink.yarn
org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster
最新发布
wuhaosheng327122的博客
03-16 675
其中一项HADOOP_HDFS_HOME写成了HADOOP_HDFS_HOM。该问题查看配置文件yarn-site.xml。改正即无报错程序正常执行。经过排查40分钟发现。
flink on yarn 提交任务报错 —— Could not deploy Yarn job cluster
一纸春秋的博客
01-18 4284
这里到项目路径里面查看一下打包好的jar包 打包之后的jar包如果是这两个，然后提交jar包的时候提交的flink-1.0-SNAPSHOT-jar-with-dependencies.jar 这个jar包，那么就会报上面的错误，这里应该是jar包里面的依赖出现了问题 我的解决措施是，直接重新创建了一个项目 然后是项目的整体pom.xml文件 <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.a.
Flink中 报错 Job execution failed
gym02的博客
04-30 8942
org.apache.flink.runtime.client.JobExecutionException: Job execution failed 可能导致错误的原因有两点： 没有在对应机器上启动 nc -lk 9999，或端口号不一致。 发送的数据类型与处理数据方法中的类型不一致。 ...
在flink集群中运行jar包出ClusterRetrieveException: Couldn‘t retrieve Yarn cluster问题的解决办法
onthe_wing的博客
08-30 3538
如果将idea上的flink代码打包运行在flink集群上，出现如下问题 有两方面需要考虑： 一、修改yarn-site.xml配置文件，原因是可能内存超过虚拟内存的限制，所以需要对yarn进行虚拟内存限制修正，将如下两个配置改为false <property> //pmem指的是默认检查物理内存,容器使用的物理内存不能超过我们限定的内存大小,因为我们上面设置了所有容器能够使用的最大内存数量,超出这个内存限制,任务就会被kill掉 <name>yarn.nodema
Flink提交时：Could not find a file system implementation for scheme ‘hdfs‘
我的博客
03-18 2000
命令和错误日志： [hadoop@localhost ~]$ flink run -c WordCountSourceDataStreamYarn ~/IdeaProjects/FlinkStudy/target/FlinkStudy-1.0-SNAPSHOT.jar ------------------------------------------------------------ The program finished with the following exception: org.ap
The main() method
iammew的专栏
03-28 277
C# programs start execution at a method named Main(). This must be a static method of a class (or struct), and must have a return type of either int or void. Although it is common to specify the pub
ProgramInvocationException: The main method caused an error: Unable to instantiate java compiler
weixin_44904163的博客
02-21 2847
背景 在 Flink集群 运行jar包时，抛出异常 Flink run -m yarn-cluster xxx.jar 解决办法 由于jar包冲突，需要去除掉自己打的 jar 包，中的部分依赖 像下面这样子，都过滤掉，不知道具体过滤哪些的话，可以看你 flink\ lib 目录下 ...
使用了flink官方示例，尽然提交任务后报错了
github_56967035的博客
06-26 1万+
使用了flink官方示例，尽然提交任务后报错了 背景：参考flink的官方示例，开发FlinkSQL的ETL应用。ETL应用从kafka消费数据，经过数据处理，将处理后的结果数据下沉存储到Clickhouse中。 问题：工程的pom文件中明明依赖了相关连接器,本地运行正常。但打包提交应用到flink时，却报错确少连接器依赖。 错误日志 当pom.xml中依赖的Kafka连接器依赖放置在Clickhouse连接器前面,报错下面日志： The program finished with the follo
Flink on Yarn 问题排查(rest.port与rest.bind-port 端口问题)
永青技术博客
04-04 6407
一、问题背景 最近在做行车数据实时分析，为了后续批流一体化的开发，前期先做技术铺垫。目前使用Flink作为批流一体切入方案。以下是基于yarn模式提交 flinksql job时出现端口冲突的问题 二、问题复述 1、我目前使用的是flink-1.12.0版本。配置文件如下 flink-conf.yaml master和worker配置 vi master bj-pan.com-04:11057 vi worker bj-pan.com-02 bj-pan.com-03...
【Flink】Flink 报错 exited with a non-zero exit code 1
九师兄
08-31 1899
1.背景 安装好集群后运行hadoop-mapreduce-examples.jar 报错，比对cdh配置后，发现少了yarn.application.classpath，conf目录一定要配置，否则也会出现以上错误。 yarn-site.xml添加 <property> <name>yarn.application.classpath</name> <value> /etc/hadoop/conf, .
redis集群java实践问题记录
Y_FQY的博客
06-05 2372
一.使用条件 redis 3.2.100 64bit java相关jar包： jedis-2.9.0 commons-pool2-2.4.2 二.问题现象以及解决方法 2.1 问题描述：Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/commons/pool2/impl/Gener
Container exited with a non-zero exit code 1. Error file: prelaunch.err.org.apache.hadoop.mapreduce.
热门推荐
colby_chenlun的博客
05-31 2万+
这两天闲的慌，装了一下新版本的HADOOP 3.2.1，的HA，装好之后，测试wordcount程序时报下面的错误 For more detailed output, check the application tracking page: http://hadoop102:8088/cluster/app/application_1590917926429_0002 Then click on links to logs of each attempt. . Failing the applicati
解决Flink on Yarn中NoClassDefFoundError问题
针对Flink on YARN出现此类依赖问题，可以采取以下步骤进行排查和解决： 1. 确认依赖版本：首先需要确认项目中jersey相关的依赖版本是否与Flink版本兼容。在给定的文件信息中，涉及到了以下jar文件： - flink-...
评论 4
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值