kafka控制台模拟消费_Flink初试——对接Kafka

本篇文章我们用 Flink Kafka Connector对接Kafka，实现一个简单的报警业务。我们暂时不去谈论理论，先上手实现这个简单的需求。

flink-connector-kafka是 flink 内置的Kafka连接器，包含了从topic读取数据的Flink Kafka Consumer 和 向topic写入数据的flink kafka producer，除了基本功能外还提供了基于 checkpoint 机制提供了完美的容错能力。

本文基于flink 1.10.1 和 flink-connector-kafka-0.10_2.11版本，pom如下：

<dependency>            <groupId>org.apache.flinkgroupId>            <artifactId>flink-connector-kafka-0.10_2.11artifactId>            <version>1.10.0versio>dependency>

以企业常见的预警业务为例，本文要实现的业务逻辑很简单，当设备上报的油桶余量不足10%时，便生成一个报警，这里我们将报警写入MySQL，以供web业务端展示报警报表。

首先我们用网络数据调试器向网关模拟发送数据，网关会将数据解析后写入kafka

kafka-console-consumer --bootstrap-server cdh1.macro.com:9092,cdh2.macro.com:9092,cdh3.macro.com:9092 --from-beginning --topic fill{
    "addTime":1593147840000,"currentAmount":0.3,"devId":"XT365-000170","devStatus":"1","ifOffline":"1","ip":"127.0.0.1","leftTankAmount":5,"realTotalAmount":2377.39,"registerTime":1606658457000,"settingAmount":0.3,"tankCapacity":1000,"totalAmount":2017.9315}{
    "addTime":1593147840000,"currentAmount":0.3,"devId":"XT365-000170","devStatus":"1","ifOffline":"1","ip":"127.0.0.1","leftTankAmount":5,"realTotalAmount":2377.69,"registerTime":1606658458000,"settingAmount":0.3,"tankCapacity":1000,"totalAmount":2017.9315}^C20/11/29 23:26:55 INFO internals.ConsumerCoordinator: [Consumer clientId=consumer-console-consumer-82199-1, groupId=console-consumer-82199] Revoke previously assigned partitions fill-020/11/29 23:26:55 INFO internals.AbstractCoordinator: [Consumer clientId=consumer-console-consumer-82199-1, groupId=console-consumer-82199] Member consumer-console-consumer-82199-1-aa5fc2e6-1f06-4714-9d89-fe080a9400e2 sending LeaveGroup request to coordinator cdh2.macro.com:9092 (id: 2147483598 rack: null) due to the consumer is being closedProcessed a total of 1200 messages

可以看到我们已经向kafka生产了1200条数据了

接下来我们写一段代码来消费kafka数据，并将报警结果写入MySQL

import com.alibaba.fastjson.JSONObject;import com.iiot.bean.InSufficient;import com.iiot.commCommon.Fill;import com.iiot.jdbc.MySQLSinks;import org.apache.flink.api.common.functi