nifi 读取mysql_使用NIFI增量获取重复记录，(Duplicate records ingesting in incremental fetch using NIFI,)...

使用NIFI增量获取重复记录，(Duplicate records ingesting in incremental fetch using NIFI,)

我正在设计一个工作流，以使用NIFI获取增量获取，源和目标数据库都在MySQL中。 处理器是QueryDatabaseTable和spiltAvro和convertAvroToJson以及convertJsonToSQL和PutSQL如下图

querydatabasetable的配置参数如下

现在源数据库表只有200条记录，当我启动进程时，流重复地摄取相同的记录。

我已经将最大值列设置为createTime ，它是时间戳并且它的值是

2017-12-07 18:48:23

为所有200条记录。

我也用ID替换了最大值列 ，但也导致了同样的问题。 什么可能导致这种重复摄取？

I am designing a workflow to get an incremental fetch using NIFI the source and target databases are in MySQL. the processors are QueryDatabaseTable and spiltAvro and convertAvroToJson and convertJsonToSQL and PutSQL as below image

and the configuration parameters for querydatabasetable is as follows

now the source database table has only 200 records, when I start the process the flow is ingesting the same records repeatedly.

I have set the maximum-value columns to createTime which is timestamp and the value for it is

2017-12-07 18:48:23

for all the 200 records.

I also replaced maximum-value columns with ID but that also resulted in same issue. What could be causing this repeated ingestion?

原文：https://stackoverflow.com/questions/47697393

2020-03-02 18:48

满意答案

根据@mattyb在NiFi中返回空值的covertJSONtoSQL中的建议，我用PutDatabaseRecord处理器替换了spiltAvr