问题来源
Flink本地提交任务运行正常,然后打包成jar在远程的Flink上运行失败。
注意:
打包的jar不能通过java -jar 运行,即使你的jar和Flink在一台机器上。
例如
Flink安装在E:\software\flink-1.7.2-bin-scala_2.12\flink-1.7.2,
jar放在E:\software\flink-1.7.2-bin-scala_2.12\flink-1.7.2\examples\streaming
需要先进入Windows cmd
E:\software\flink-1.7.2-bin-scala_2.12\flink-1.7.2\bin>
然后执行命令
flink.bat run …\examples\streaming\FlinkRemoteDemoJar-1.0-SNAPSHOT-jar-with-dependencies.jar --bootstrap.servers a.b.c.d:9092 --limitEnabled 2 --timeLimit 2 --group.id grp01 --nodeId a1b2c34
症状
无论是通过命令行还是web界面都报如下错误
Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.commons.collections.map.LinkedMap to field org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.pendingOffsetsToCommit of type org.apache.commons.collections.map.LinkedMap in instance of org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer
也就是
“LinkedMap cannot be cast to LinkedMap exceptions”.
截图
解决办法
在c/onf/flink-conf.yaml 添加如下内容并重启 flink.
classloader.resolve-order: parent-first
本质原因
LinkedMap class is being loaded from two different packages, and those are being assigned to each other.
官方文档
https://ci.apache.org/projects/flink/flink-docs-release-1.8/monitoring/debugging_classloading.html