flink cdc同步Oracle数据库资料到Doris问题集锦

问题1：flink 1.14 包 org.apache.flink.shaded.guava 版本冲突

java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava18/com/google/common/util/concurrent/ThreadFactoryBuilder at com.ververica.cdc.debezium.DebeziumSourceFunction.open(DebeziumSourceFunction.java:218) ~[flink-connector-debezium-2.2.0.jar:2.2.0] at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34) ~[flink-core-1.14.2.jar:1.14.2] at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:100) ~[flink-streaming-java_2.12-1.14.2.jar:1.14.2] at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:110) ~[flink-streaming-java_2.12-1.14.2.jar:1.14.2] at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711) ~[flink-streaming-java_2.12-1.14.2.jar:1.14.2] at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.call(StreamTaskActionExecutor.java:100) ~[flink-streaming-java_2.12-1.14.2.jar:1.14.2] at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687) ~[flink-streaming-java_2.12-1.14.2.jar:1.14.2] at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654) ~[flink-streaming-java_2.12-1.14.2.jar:1.14.2] at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[flink-runtime-1.14.2.jar:1.14.2] at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) ~[flink-runtime-1.14.2.jar:1.14.2] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) ~[flink-runtime-1.14.2.jar:1.14.2] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) ~[flink-runtime-1.14.2.jar:1.14.2] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_321]

根据网友建议，重新引入flink-shaded-guava，重新制定冲突依赖

<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-shaded-guava</artifactId> <version>18.0-13.0</version> </dependency>

继续报异常

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava30/com/google/common/collect/Lists at org.apache.flink.streaming.api.transformations.LegacySinkTransformation.getTransitivePredecessors(LegacySinkTransformation.java:117) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.lambda$existsUnboundedSource$1(StreamGraphGenerator.java:481) at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361) at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:449) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.existsUnboundedSource(StreamGraphGenerator.java:478) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.shouldExecuteInBatchMode(StreamGraphGenerator.java:459) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.generate(StreamGraphGenerator.java:309) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2083) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2069) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1899) at com.zsfs.hz.task.Oracle2DorisFlinkCDCTest.main(Oracle2DorisFlinkCDCTest.java:56)

最后解决方式：

一开始flink使用的是1.14.2，flink-connector-jdbc版本也是1.14.2；这两个依赖中使用的flink-shaded-guava版本不一致。flink使用的是guava30，connect使用的是guava18。我这边将flink版本降成1.13.6，然后问题就解决了

备注：目前flink14和flink15没有怎么解决这个问题，带后续解决后，再升级flink集群