本文环境
-
Maven 3.6.3
-
Java 8
Flink Kafka连接器
Maven引入相关依赖:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>1.9.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.11</artifactId>
<version>1.9.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_2.11</artifactId>
<version>1.9.0</version>
</dependency>
Kafka消费端JAVA:
// 1.获取flink流计算的运行环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//Kafka props
Properties properties = new Properties();
//指定Kafka的Broker地址
properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,brokers);
//指定组ID
properties.put(ConsumerConfig.GROUP_ID_CONFIG,TrackConstant.GROUP_ID);
//如果没有记录偏移量,第一次从最开始消费
properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
// 2.从kafka读取数据
FlinkKafkaConsumer<String> kafkaSource = new FlinkKafkaConsumer<>("test-flink-topic", new SimpleStringSchema(), properties);
DataStreamSource<String> stringDataStreamSource = env.addSource(kafkaSource);
//3.调用Sink
stringDataStreamSource.print();
//4.启动流计算
env.execute("KafkaSouceReview");
相关资料
Flink官方概述及原理:
https://nightlies.apache.org/flink/flink-docs-release-1.9/zh/concepts/runtime.html
Kafka连接器官方文档:
https://nightlies.apache.org/flink/flink-docs-release-1.9/zh/dev/connectors/kafka.html