org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException:

使用nifi+kylo报错了。 

2019-03-05 15:03:08,226 ERROR [Timer-Driven Process Thread-8] c.t.nifi.v2.thrift.ExecuteHQL ExecuteHQL[id=4822488d-3004-39ad-0e1b-2aaafa7edf73] ExecuteHQL[id=4822488d-3004-39ad-0e1b-2aaafa7edf73] failed to process due to org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException: Not in union ["null","float"]: 67.9; rolling back session: {}
org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException: Not in union ["null","float"]: 67.9
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296)
at com.thinkbiganalytics.util.JdbcCommon.convertToAvroStream(JdbcCommon.java:142)
at com.thinkbiganalytics.nifi.v2.thrift.ExecuteHQL$1.process(ExecuteHQL.java:168)
at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2571)
at com.thinkbiganalytics.nifi.v2.thrift.ExecuteHQL.onTrigger(ExecuteHQL.java:162)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.avro.UnresolvedUnionException: Not in union ["null","float"]: 67.9
at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:604)
at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
... 16 common frames omitted
2019-03-05 15:03:08,226 ERROR [Timer-Driven Process Thread-8] c.t.nifi.v2.thrift.ExecuteHQL ExecuteHQL[id=4822488d-3004-39ad-0e1b-2aaafa7edf73] ExecuteHQL[id=4822488d-3004-39ad-0e1b-2aaafa7edf73] failed to process session due to org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException: Not in union ["null","float"]: 67.9: {}
org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException: Not in union ["null","float"]: 67.9
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296)
at com.thinkbiganalytics.util.JdbcCommon.convertToAvroStream(JdbcCommon.java:142)
at com.thinkbiganalytics.nifi.v2.thrift.ExecuteHQL$1.process(ExecuteHQL.java:168)
at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2571)
at com.thinkbiganalytics.nifi.v2.thrift.ExecuteHQL.onTrigger(ExecuteHQL.java:162)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.avro.UnresolvedUnionException: Not in union ["null","float"]: 67.9
at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:604)
at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
... 16 common frames omitted
2019-03-05 15:03:08,226 WARN [Timer-Driven Process Thread-8] c.t.nifi.v2.thrift.ExecuteHQL ExecuteHQL[id=4822488d-3004-39ad-0e1b-2aaafa7edf73] Processor Administratively Yielded for 1 sec due to processing failure
2019-03-05 15:03:08,226 WARN [Timer-Driven Process Thread-8] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding ExecuteHQL[id=4822488d-3004-39ad-0e1b-2aaafa7edf73] due to uncaught Exception: org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException: Not in union ["null","float"]: 67.9
2019-03-05 15:03:08,226 WARN [Timer-Driven Process Thread-8] o.a.n.c.t.ContinuallyRunProcessorTask 
org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException: Not in union ["null","float"]: 67.9
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296)
at com.thinkbiganalytics.util.JdbcCommon.convertToAvroStream(JdbcCommon.java:142)
at com.thinkbiganalytics.nifi.v2.thrift.ExecuteHQL$1.process(ExecuteHQL.java:168)
at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2571)
at com.thinkbiganalytics.nifi.v2.thrift.ExecuteHQL.onTrigger(ExecuteHQL.java:162)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.avro.UnresolvedUnionException: Not in union ["null","float"]: 67.9
at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:604)
at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
... 16 common frames omitted
2019-03-05 15:03:08,364 INFO [NiFi Web Server-521] o.a.n.c.s.StandardProcessScheduler Stopping ExecuteHQL[id=4822488d-3004-39ad-0e1b-2aaafa7edf73]
2019-03-05 15:03:08,364 INFO [NiFi Web Server-521] o.a.n.controller.StandardProcessorNode Stopping processor: class com.thinkbiganalytics.nifi.v2.thrift.ExecuteHQL
2019-03-05 15:03:08,364 INFO [StandardProcessScheduler Thread-7] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling ExecuteHQL[id=4822488d-3004-39ad-0e1b-2aaafa7edf73] to run
2019-03-05 15:03:08,421 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@29e598c5 // Another save pending = true
2019-03-05 15:03:09,458 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@29e598c5 // Another save pending = false

@Matt Burgess

JDBC spec does state that REAL maps to (single-precision) float; FLOAT maps to double precision, thus the mapping to in avro should be to double not float, reference table B-1 page 190:

http://download.oracle.com/otn-pub/jcp/jdbc-4_3-mrel3-eval-spec/jdbc4.3-fr-spec.pdf

So the following code fix resolved the issue.

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java#L527

original

case FLOAT:
case REAL:                  
builder.name(columnName).type().unionOf().nullBuilder().endNull().and().floatType().endUnion().noDefault();                    
break;
              
case DOUBLE:                   
builder.name(columnName).type().unionOf().nullBuilder().endNull().and().doubleType().endUnion().noDefault();                    
break;
Modified

case REAL:                  
builder.name(columnName).type().unionOf().nullBuilder().endNull().and().floatType().endUnion().noDefault();                    
break;
 
case FLOAT:        
case DOUBLE:                   
builder.name(columnName).type().unionOf().nullBuilder().endNull().and().doubleType().endUnion().noDefault();                    
break;

在这儿找到的：

https://community.hortonworks.com/questions/144420/apache-nifi-executesql-processor-error-orgapacheav.html

 

亲测可以使用，这是kylo的一个bug