Flink1.9整合Kafka

本文环境

• Maven 3.6.3

• Java 8

Flink Kafka连接器

Maven引入相关依赖:

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-java</artifactId>
  <version>1.9.0</version>
  <scope>provided</scope>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-streaming-java_2.11</artifactId>
  <version>1.9.0</version>
  <scope>provided</scope>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-connector-kafka_2.11</artifactId>
  <version>1.9.0</version>
</dependency>

Kafka消费端JAVA:

// 1.获取flink流计算的运行环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

//Kafka props
Properties properties = new Properties();
//指定Kafka的Broker地址
properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,brokers);
//指定组ID
properties.put(ConsumerConfig.GROUP_ID_CONFIG,TrackConstant.GROUP_ID);
//如果没有记录偏移量，第一次从最开始消费
properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

// 2.从kafka读取数据
FlinkKafkaConsumer<String> kafkaSource = new FlinkKafkaConsumer<>("test-flink-topic", new SimpleStringSchema(), properties);

DataStreamSource<String> stringDataStreamSource = env.addSource(kafkaSource);

//3.调用Sink
stringDataStreamSource.print();

//4.启动流计算

env.execute("KafkaSouceReview");

相关资料

Flink官方概述及原理：

https://nightlies.apache.org/flink/flink-docs-release-1.9/zh/concepts/runtime.html

Kafka连接器官方文档：

https://nightlies.apache.org/flink/flink-docs-release-1.9/zh/dev/connectors/kafka.html