Flink On Yarn Pre-job提交空指针报错

背景

通过命令提交job到yarn是正常运行的，命令如下：

./bin/flink run -d -m yarn-cluster ./examples/streaming/TopSpeedWindowing.jar

项目需求通过代码的形式提交，这边方便获取jobId和applicationId，如是找了一个开源的代码，代码地址flink-spark-submiter，倒腾了半天终于可以提交了，执行如下：

java -jar flink-yarn-submiter-1.0-SNAPSHOT.jar -hadoopConfDir /root/app/hadoop/etc/hadoop -name test-one -flinkConfDir /opt/topology/service/flink/conf -flinkJarPath /opt/topology/service/flink/lib  -runJarPath /opt/topology/service/flink/examples/streaming/TopSpeedWindowing.jar

yarn也收到提交申请，如图：

yarn这块状态从一开始的ACCEPTED变为RUNNING，但是过会儿就失败了，查看日志Flink集群启动失败，具体错误如下：

2022-02-25 09:20:03,186 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Fatal error occurred in the cluster entrypoint.
org.apache.flink.util.FlinkException: JobMaster for job 1071fbecc2f79b948678f19dc58652fa failed.
	at org.apache.flink.runtime.dispatcher.Dispatcher.jobMasterFailed(Dispatcher.java:910) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at org.apache.flink.runtime.dispatcher.Dispatcher.jobManagerRunnerFailed(Dispatcher.java:473) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at org.apache.flink.runtime.dispatcher.Dispatcher.handleJobManagerRunnerResult(Dispatcher.java:450) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$runJob$3(Dispatcher.java:427) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456) ~[?:1.8.0_281]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRunAsync$4(AkkaRpcActor.java:455) ~[flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) ~[flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:455) ~[flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:213) ~[flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78) ~[flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163) ~[flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.actor.Actor.aroundReceive(Actor.scala:537) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.actor.Actor.aroundReceive$(Actor.scala:535) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.actor.ActorCell.invoke(ActorCell.scala:548) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.dispatch.Mailbox.run(Mailbox.scala:231) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:243) [flink-rpc-akka_5112991e-b5cb-40c7-a40f-c4687a7b1867.jar:1.14.2]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_281]
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067) [?:1.8.0_281]
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703) [?:1.8.0_281]
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172) [?:1.8.0_281]
Caused by: org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster.
	at org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1609) ~[?:1.8.0_281]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_281]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_281]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_281]
Caused by: java.util.concurrent.CompletionException: java.lang.NullPointerException
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_281]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606) ~[?:1.8.0_281]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_281]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_281]
	at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_281]
Caused by: java.lang.NullPointerException
	at org.apache.flink.runtime.deployment.TaskDeploymentDescriptorFactory$PartitionLocationConstraint.fromJobType(TaskDeploymentDescriptorFactory.java:328) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:339) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:190) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
	at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:130) ~[flink-dist_2.11-1.14.2.jar:1.14.2]

问题排查

从截图上面看，是由于空指针导致jobMaster启动失败了，具体位置TaskDeploymentDescriptorFactory.java:328，查看源码如下：

查看类文件注意版本，版本不同类可能不一样。
再往上找看下JobType是哪里来的，如图：

可以看到是JobGraph对象下的属性，查看flink-spark-submiter里面的代码，看下是不是没有赋值，发现flink-spark-submiter里面的JobGraph对象是没有jobType这个属性的，再对比发现flink-spark-submiter用的flink版本是1.12.1，而我们部署的版本是1.14.2，升级flink-spark-submiter的flink版本后该问题得到解决。