flink 踩坑记录

最新推荐文章于 2024-07-29 02:10:34 发布

时光不在~

最新推荐文章于 2024-07-29 02:10:34 发布

阅读量1.5k

点赞数

文章标签： flink 大数据

本文链接：https://blog.csdn.net/theweather/article/details/131433563

版权

文章描述了在使用阿里云实时计算产品时遇到的两个主要问题。第一个问题是Java程序运行时抛出NoClassDefFoundError，涉及Kafka客户端的Producer类，解决方法是检查并修正类路径或依赖管理。第二个问题是在创建MySQL-CDC源时的验证异常，客服建议修改connector为mysql，但实际问题在于依赖冲突，通过将本地pom中的flink依赖设置为provided解决了问题。第三个问题涉及到Flink作业中的协调器失败，原因是类加载错误，解决办法是解决了Guava库的版本不匹配问题。

摘要由CSDN通过智能技术生成

1、 java.lang.NoClassDefFoundError: org/apache/kafka/clients/producer/Producer

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/clients/producer/Producer
	at com.hll.flinkJob.qts.QtsProductClassifyJob.main(QtsProductClassifyJob.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.clients.producer.Producer
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 1 more

2、

2023-06-28 10:25:55,775 WARN  org.apache.flink.client.deployment.application.DetachedApplicationRunner [] - Could not execute application: 
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Unable to create a source for reading table 'default_catalog.default_database.f_shop'.

Table options are:

'connector'='mysql-cdc'
'database-name'='db_shop_center'
'hostname'='host'
'password'='******'
'port'='3306'
'server-id'='5300-5309'
'table-name'='shop'
'username'='root'
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) ~[flink-dist_2.12-1.13.0.jar:1.13.0]
        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist_2.12-1.13.0.jar:1.13.0]
        at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) ~[flink-dist_2.12-1.13.0.jar:1.13.0]
        at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84) ~[flink-dist_2.12-1.13.0.jar:1.13.0]
        at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70) ~[flink-dist_2.12-1.13.0.jar:1.13.0]
        at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:102) ~[flink-dist_2.12-1.13.0.jar:1.13.0]
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) [?:1.8.0_341]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_341]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_341]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_341]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_341]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_341]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_341]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_341]
Caused by: org.apache.flink.table.api.ValidationException: Unable to create a source for reading table 'default_catalog.default_database.f_shop'.

Table options are:

'connector'='mysql-cdc'
'database-name'='db_shop_center'
'hostname'='host'
'password'='******'
'port'='3306'
'server-id'='5300-5309'
'table-name'='shop'
'username'='root'
        at org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:131) ~[flink-table_2.12-1.13.0.jar:1.13.0]
        at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.createDynamicTableSource(CatalogSourceTable.java:116) ~[flink-table-blink_2.12-1.13.0.jar:1.13.0]

在使用阿里云实时计算产品，部署job遇到的问题，咨询客服说是需要将connector改为mysql，但实例之后并没有效果，后来因为其他报错得知阿里flink上已经存在flink的相关依赖了，于是改本地pom，将所有flink依赖加上 <scope>provided</scope> 问题解决。

3、

Caused by: org.apache.flink.util.FlinkException: Global failure triggered by OperatorCoordinator for 'Source: f_material_classify[1] -> (Calc[2], Calc[4])' (operator e3dfc0d7e9ecd8a43f85f0b68ebf3b80).
	at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder$LazyInitializedCoordinatorContext.failJob(OperatorCoordinatorHolder.java:632) ~[flink-dist-1.15-vvr-6.0.6-SNAPSHOT.jar:1.15-vvr-6.0.6-SNAPSHOT]
	at org.apache.flink.runtime.operators.coordination.RecreateOnResetOperatorCoordinator$QuiesceableContext.failJob(RecreateOnResetOperatorCoordinator.java:257) ~[flink-dist-1.15-vvr-6.0.6-SNAPSHOT.jar:1.15-vvr-6.0.6-SNAPSHOT]
	at org.apache.flink.runtime.source.coordinator.SourceCoordinatorContext.failJob(SourceCoordinatorContext.java:317) ~[flink-dist-1.15-vvr-6.0.6-SNAPSHOT.jar:1.15-vvr-6.0.6-SNAPSHOT]
	at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$runInEventLoop$10(SourceCoordinator.java:495) ~[flink-dist-1.15-vvr-6.0.6-SNAPSHOT.jar:1.15-vvr-6.0.6-SNAPSHOT]
	at org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40) ~[flink-dist-1.15-vvr-6.0.6-SNAPSHOT.jar:1.15-vvr-6.0.6-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_102]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_102]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:186) ~[?:1.8.0_102]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:299) ~[?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147) ~[?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) ~[?:1.8.0_102]
	at java.lang.Thread.run(Thread.java:834) ~[?:1.8.0_102]
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/shaded/guava18/com/google/common/util/concurrent/ThreadFactoryBuilder
	at com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.startAsynchronouslySplit(MySqlSnapshotSplitAssigner.java:195) ~[?:?]
	at com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.open(MySqlSnapshotSplitAssigner.java:163) ~[?:?]
	at com.ververica.cdc.connectors.mysql.source.assigners.MySqlHybridSplitAssigner.open(MySqlHybridSplitAssigner.java:95) ~[?:?]
	at com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator.start(MySqlSourceEnumerator.java:101) ~[?:?]
	at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$start$1(SourceCoordinator.java:210) ~[flink-dist-1.15-vvr-6.0.6-SNAPSHOT.jar:1.15-vvr-6.0.6-SNAPSHOT]
	at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$runInEventLoop$10(SourceCoordinator.java:481) ~[flink-dist-1.15-vvr-6.0.6-SNAPSHOT.jar:1.15-vvr-6.0.6-SNAPSHOT]
	at org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40) ~[flink-dist-1.15-vvr-6.0.6-SNAPSHOT.jar:1.15-vvr-6.0.6-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_102]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_102]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:186) ~[?:1.8.0_102]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:299) ~[?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147) ~[?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) ~[?:1.8.0_102]
	at java.lang.Thread.run(Thread.java:834) ~[?:1.8.0_102]

这个类ThreadFactoryBuilder找不到，我们使用maven helper 看一下包依赖，发现，flink依赖使用的是flink-shaded-guava:30.1.1-jre，而mysql-cdc 2.2.1用的是guava:30.0-jre，但是这里确报的guava18的错。尝试把flink-shaded-guava-18.0-13.0.jar版本的guava放到flink服务器上，在启动就不报这个错了。完美解决，看网上有很多需要改源码再编译打包，这种方式就简单很多了。