flink-提交jar包 submit 报错未序列化

最新推荐文章于 2024-04-25 16:10:59 发布

三少说

最新推荐文章于 2024-04-25 16:10:59 发布

阅读量1w

点赞数 4

文章标签： java flink 大数据

本文链接：https://blog.csdn.net/qq_44112474/article/details/123728683

版权

问题排查同时被 2 个专栏收录

5 篇文章 0 订阅

订阅专栏

大数据体系

4 篇文章 0 订阅

订阅专栏

flink-提交jar包 submit 报错未序列化

flink-web控制台 submit jar包报错堆栈(org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute application. at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:107) at )

（看不出来什么，我们需要去 flink-server 端看运行日志）

org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute application. 
	at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:107) at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) 
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) 	
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) 
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595) 
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) 
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.CompletionException: org.apache.flink.util.FlinkRuntimeException: Could not execute application.
	 at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
	  at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) 
	  at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592) ... 7 more Caused by: org.apache.flink.util.FlinkRuntimeException: Could not execute application. 
	  at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:88)
	   at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) 
	   at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:102) 
	   at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ... 7 more Caused by: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: com.jdd.martech.export.kafka.config.KafkaConfig@3a786b17 is not serializable. The object probably contains or references non serializable fields. 
	   at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:360) 
	   at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:213) 
	   at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) 
	   at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ... 10 more Caused by: org.apache.flink.api.common.InvalidProgramException: com.jdd.martech.export.kafka.config.KafkaConfig@3a786b17 is not serializable. The object probably contains or references non serializable fields. at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:164) at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132) at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132) at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:2000) at org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:203) at org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1243) at com.jd.jr.martech.data.main.ExecuteMain.main(ExecuteMain.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:343) ... 13 more Caused by: java.io.NotSerializableException: com.jdd.martech.export.kafka.config.KafkaConfig at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:624) at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:143) ... 25 more

flink-server 服务端日志报错如下：(org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: com.jdd.martech.export.kafka.config.KafkaConfig@3a786b17 is not serializable. The object probably contains or references non serializable fields.)


2022-03-25 09:50:35,392 WARN  org.apache.flink.client.deployment.application.DetachedApplicationRunner [] - Could not execute application:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: com.jdd.martech.export.kafka.config.KafkaConfig@3a786b17 is not serializable. The object probably contains or references non serializable fields.
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:360) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:213) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:102) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) [?:1.8.0_91]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_91]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_91]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_91]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: org.apache.flink.api.common.InvalidProgramException: com.jdd.martech.export.kafka.config.KafkaConfig@3a786b17 is not serializable. The object probably contains or references non serializable fields.
        at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:164) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:2000) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:203) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1243) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at com.jd.jr.martech.data.main.ExecuteMain.main(ExecuteMain.java:83) ~[?:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_91]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_91]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_91]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_91]
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:343) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        ... 13 more
Caused by: java.io.NotSerializableException: com.jdd.martech.export.kafka.config.KafkaConfig
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) ~[?:1.8.0_91]
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) ~[?:1.8.0_91]
        at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:624) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:143) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:2000) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:203) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1243) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        at com.jd.jr.martech.data.main.ExecuteMain.main(ExecuteMain.java:83) ~[?:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_91]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_91]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_91]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_91]
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:343) ~[flink-dist_2.11-1.12.1.jar:1.12.1]
        ... 13 more
2022-03-25 09:50:35,393 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler   [] - Exception occurred in REST handler: Could not execute application.

观看日志可以发现是我们的业务类未序列化

flink业务类报错

仔细研究发现

如果继承了flink的算子类，构造方法中涉及到的所有类（包括子类，引入的包）必须序列化，如果不能序列化，请不要放到构造方法中

出现这个原因我的理解：flink 需要先拿到算子直接的关系构建有向无环图，所以需要调用算子的构造方法

三少说

关注

4
点赞
踩
0

收藏

觉得还不错? 一键收藏
5
评论
flink-提交jar包 submit 报错未序列化

flink-提交jar包 submit 报错未序列化flink-web控制台 submit jar包报错堆栈(org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute application. at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:107)
复制链接

扫一扫