flinksql 监控kafka topic 是否发数据过来

fyl005

已于 2022-08-29 11:06:51 修改

阅读量1k

点赞数 2

分类专栏： flink kafka streamx 文章标签： kafka 大数据 flink

于 2022-08-29 10:59:16 首次发布

本文链接：https://blog.csdn.net/zuixue_lei/article/details/126580649

版权

streamx 同时被 3 个专栏收录

6 篇文章 0 订阅

订阅专栏

kafka

5 篇文章 0 订阅

订阅专栏

flink

3 篇文章 0 订阅

订阅专栏

1、streamx 中代码：

-- streamx 执行

-- 读取kafka的数据并创建表
--source topic : stg_traffic_participants
CREATE  TABLE `stg_traffic_participants_kafka` (
    `ts` TIMESTAMP(3) METADATA FROM 'timestamp'
) WITH (
        'connector' = 'kafka',
        'format' = 'json',
        'topic' = 'ihs_v2x_tgac_vhc_rt',
        'scan.startup.mode' = 'latest-offset',
        'properties.bootstrap.servers' = 'ip:port',
        'properties.group.id' = 'ihs_v2x_tgac_vhc_rt'
 );

--sink kafka
--sink topic : ogl_ivis_tfc_partcp_rt
CREATE  TABLE `test` (
    `ts` TIMESTAMP(3) METADATA FROM 'timestamp'
) WITH (
        'connector' = 'kafka',
        'format' = 'json',
        'topic' = 'test',
        'properties.bootstrap.servers' = 'ip:port',
         'sink.partitioner'='round-robin'
 );

insert into test
select
    ts as ts
from stg_traffic_participants_kafka
;

2、当有数据到来时，会有报错信息，从报错信息中可以看到 topic 中的数据格式。

java.io.IOException: Failed to deserialize JSON '[{"alarmCode":"00000000000000000000000000000011","derection":"187","encrypt":"1","gpsTime":"2022-08-29 10:22:40","lat":"39.511066","lon":"115.997756","stateCode":"00000000000000000000000000000000","vec1":"18","vec2":"18","vehicleColor":"1","vehicleNo":"京KHM007"}]'.
	at org.apache.flink.formats.json.JsonRowDataDeserializationSchema.deserialize(JsonRowDataDeserializationSchema.java:105)
	at org.apache.flink.formats.json.JsonRowDataDeserializationSchema.deserialize(JsonRowDataDeserializationSchema.java:46)
	at org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:82)
	at org.apache.flink.streaming.connectors.kafka.table.DynamicKafkaDeserializationSchema.deserialize(DynamicKafkaDeserializationSchema.java:130)
	at org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:179)
	at org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.runFetchLoop(KafkaFetcher.java:142)
	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:826)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
	at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:263)
Caused by: java.lang.ClassCastException: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ArrayNode cannot be cast to org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode
	at org.apache.flink.formats.json.JsonToRowDataConverters.lambda$createRowConverter$ef66fe9a$1(JsonToRowDataConverters.java:343)
	at org.apache.flink.formats.json.JsonToRowDataConverters.lambda$wrapIntoNullableConverter$de0b9253$1(JsonToRowDataConverters.java:375)
	at org.apache.flink.formats.json.JsonRowDataDeserializationSchema.deserialize(JsonRowDataDeserializationSchema.java:99)
	... 9 more