Flink执行报错<很晦涩>

org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator
	at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:658)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:616)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:596)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708)
	at org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:53)
	at com.demo.flink.base.transform.EffectiveFlatMapFunc.flatMap(EffectiveFlatMapFunc.java:59)
	at com.demo.flink.base.transform.EffectiveFlatMapFunc.flatMap(EffectiveFlatMapFunc.java:23)
	at org.apache.flink.streaming.api.operators.StreamFlatMap.processElement(StreamFlatMap.java:50)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:641)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:616)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:596)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:730)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:708)
	at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
	at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:111)
	at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:398)
	at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:91)
	at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:156)
	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:718)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
	at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:200)
Caused by: java.lang.NullPointerException
	at com.demo.flink.base.transform.SideFlatMapFunc.flatMap(SideFlatMapFunc.java:64)
	at com.demo.flink.base.transform.SideFlatMapFunc.flatMap(SideFlatMapFunc.java:24)
	at org.apache.flink.streaming.api.operators.StreamFlatMap.processElement(StreamFlatMap.java:50)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:641)
	... 22 more

最下边给出了明确的报错信息,NullPointerException。但是按照执行逻辑,不应该报错才对。顺着这个思路往上一个算子找。发现从ObjectNode中get到值后,没有进行asText转换,导致与String进行比较时,始终是false。

最开始被最上面的逻辑所迷惑 Could not forward element to next operator,很多文章说“配置的Watermark中的timestamp有问题,需要指定一下你的输入数据中时间的格式,比如是"2018-02-03 21:00:00",格式应该配置成"yyyy-MM-dd HH:mm:ss"”,但是我觉得这个可能不是关键问题,因为Flink不会强制你为每条消息绑定watermark。

  • 2
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 2
    评论
要将Flink的KafkaSource<ObjectNode>转换为DataStream<RowData>,需要进行以下步骤: 1. 创建一个DeserializationSchema,将ObjectNode反序列化为RowData对象。例如,可以使用JsonRowDataDeserializationSchema。 2. 使用KafkaSource从Kafka主题中读取ObjectNode对象,并使用上一步中创建的DeserializationSchema将其转换为RowData对象。 3. 在DataStream上应用map操作,将RowData对象转换为所需的格式。 下面是一个示例代码,使用JsonRowDataDeserializationSchema将ObjectNode转换为RowData,然后将其转换为Tuple2<String, String>: ``` import org.apache.flink.api.common.serialization.DeserializationSchema; import org.apache.flink.api.common.serialization.SimpleStringSchema; import org.apache.flink.api.common.typeinfo.TypeInformation; import org.apache.flink.api.java.tuple.Tuple2; import org.apache.flink.api.java.typeutils.RowTypeInfo; import org.apache.flink.formats.json.JsonRowDataDeserializationSchema; import org.apache.flink.streaming.api.datastream.DataStream; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer; import org.apache.flink.table.api.TableSchema; import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; import org.apache.flink.table.types.logical.RowType; import org.apache.flink.types.Row; import com.fasterxml.jackson.databind.node.ObjectNode; import java.util.Properties; public class KafkaObjectNodeToRowData { public static void main(String[] args) throws Exception { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); StreamTableEnvironment tEnv = StreamTableEnvironment.create(env); Properties properties = new Properties(); properties.setProperty("bootstrap.servers", "localhost:9092"); properties.setProperty("group.id", "test"); // create a DeserializationSchema to deserialize ObjectNode to RowData RowType rowType = new RowType( new TableSchema.Builder() .field("id", DataTypes.BIGINT()) .field("name", DataTypes.STRING()) .build().toRowDataType().getChildren()); DeserializationSchema<RowData> deserializationSchema = new JsonRowDataDeserializationSchema(rowType); // create a FlinkKafkaConsumer to read ObjectNode from Kafka FlinkKafkaConsumer<ObjectNode> kafkaConsumer = new FlinkKafkaConsumer<>("my-topic", deserializationSchema, properties); // read from Kafka and convert ObjectNode to RowData DataStream<RowData> rowDataStream = env.addSource(kafkaConsumer) .map(objectNode -> { // convert ObjectNode to RowData Row row = new Row(objectNode.size()); for (int i = 0; i < objectNode.size(); i++) { row.setField(i, objectNode.get(i).asText()); } return row; }) .returns(new RowTypeInfo(rowType.getFields().stream().map(RowType.RowField::getType).toArray(TypeInformation[]::new), rowType.getFieldNames().toArray(new String[0]))); // convert RowData to Tuple2<String, String> DataStream<Tuple2<String, String>> tupleStream = rowDataStream .map(rowData -> Tuple2.of(rowData.getField(0).toString(), rowData.getField(1).toString())) .returns(Types.TUPLE(Types.STRING, Types.STRING)); // do something with the tupleStream tupleStream.print(); env.execute("KafkaObjectNodeToRowData"); } } ```

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 2
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值