flink kafka connect mysql 广播流 BroadcastStream 错误解决记录 Not all required tasks are currently running.
错误如图
错误文字
[Checkpoint Timer] INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Failed to trigger checkpoint for job 4efe5a68da3ab807288aa4d13e236ee5 because Some tasks of the job have already finished and checkpointing with finished tasks is not enabled. Failure reason: Not all required tasks are currently running.
错误原因
flink-kakfa流关联mysql数据行成的flink广播流 BroadcastStream报错是因为flink在checkpoint时会去检查flink-kakfa流和flink-mysql广播流的流状态,因mysql 我这里用的是org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromCollection(java.util.Collection)导入的数据,很明显是一个批处理程序,在检查时已经变成finish状态,因此报错( Not all required tasks are currently running.),checkpoint 成功的前提下是流状态都要是 running 状态。
解决方案
这里要让mysql广播流永远running,可以有以下方案
1、自定义source,在run方法里面永远循环
2、将mysql维度表数据写入kafka,把这个kafka流弄成广播流
具体方案后面写