org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster.

在cdh集群上,通过如下命令,提交flink任务到yarn集群中:

flink run -m yarn-cluster -d -ynm my_flink_job xxx.jar

报错,信息如下:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Could not deploy Yarn job cluster.

Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: Could not deploy Yarn job cluster.

Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment.

Diagnostics from YARN: Application application_1640140324841_0003 failed 1 times (global limit =4; local limit is =1) due to AM Container for appattempt_1640140324841_0003_000001 exited with  exitCode: 1
Failing this attempt.Diagnostics: [2021-12-22 11:23:34.422]Exception from container-launch.
Container id: container_e44_1640140324841_0003_01_000001
Exit code: 1
Shell output: main : command provided 1
main : run as user is etl_admin
main : requested yarn user is etl_admin
Getting exit code file…
Creating script paths…
Writing pid file…
Writing to tmp file /data1/yarn/nm/nmPrivate/application_1640140324841_0003/container_e44_1640140324841_0003_01_000001/container_e44_1640140324841_0003_01_000001.pid.tmp
Writing to cgroup task files…
Creating local dirs…
Launching container…

解决方法:

Look is the flink version of the idea is 1.11.0, the flink version on the cluster is 1.13.1
Directly upgrade the flink version in the idea to 1.13.1, done!

引用: Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed. Caused by: java.lang.Exception: java.net.SocketException: Connection reset Caused by: java.net.SocketException: Connection reset。 引用: 3.当设置的分区数多于机器的CPU数会发生数据混乱的错误,导致计算不正确。本身机器的CPU为4核。 org.apache.flink.runtime.entrypoint.ClusterEntrypointException: Failed to initialize the cluster entrypoint YarnSessionClusterEntrypoint. at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:501) at org.apache.flink.yarn.entrypoint.YarnSessionClusterEntrypoint.main(YarnSessionClusterEntrypoint.java:93) Caused by: java.net.ConnectException: Call From node2/192.168.40.62 to node1:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)。 引用: 原因:socket连接重置,可能是使用不同的方式或者是重复提交flink任务,导致socket端口占用导致。2.No new data sinks have been defined since。原因:未被定义的数据输出 flink的批处理不需要行动算子来触发,因此删除最后一行的 //启动流式处理,如果没有该行代码上面的程序不会运行 streamEnv.execute("wordcount")。 "Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException: Job execution failed." 这个错误是由于flink作业执行失败所引发的异常。可能的原因是网络连接重置或未定义的数据输出。对于网络连接重置,可以检查是否使用了不同的连接方式或是否重复提交了flink任务。对于未定义的数据输出,可以检查是否没有定义数据的输出操作。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

赶路人儿

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值