Flink--初识 DataStream Connector Kafka

• Flink 作为比较适合流式处理的计算框架，在流式处理当中，比较搭配的消息中间件为 Kafka
• 本次使用的 Kafka 版本为 2.1.0-cdh6.2.0
• Flink 版本为 1.11.2，Scala 版本为 2.12.10

官网

Flink provides an Apache Kafka connector for reading data from and writing data to Kafka topics with exactly-once guaruntees.

1. 增加 POM 文件

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kafka_2.12</artifactId>
    <version>1.11.2</version>
</dependency>

2. 使用 Kafka 作为 Source

2.1 创建 Topic

[root@bigdatatest03 bin]# ./kafka-topics.sh --create --zookeeper  bigdatatest02:2181,bigdatatest02:2181,bigdatatest03:2181 --partitions 3 --replication-factor 3 --topic flink_kafka_source

• 创建生产者

[root@bigdatatest03 bin]# ./kafka-console-producer.sh \
> --broker-list bigdatatest01:9092,bigdatatest02:9092,bigdatatest03:9092 \
> --topic flink_kafka_source

• 创建消费者

[root@bigdatatest03 bin]# ./kafka-console-consumer.sh \
> --bootstrap-server bigdatatest01:9092,bigdatatest02:9092,bigdatatest03:9092 \
> --topic flink_kafka_source \
> --from-beginning

2.2 Code

package com.xk.bigdata.flink.datastream.connector

import java.util.Properties

import org.apache.flink.api.common.serialization.SimpleStringSchema
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer

/**
 * 使用 Kafka 作为数据源
 */
object KafkaSourceApp {

  def main(args: Array[String]): Unit = {
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    val properties = new Properties()
    val bootStrap = "bigdatatest02:9092,bigdatatest03:9092,bigdatatest04:9092"
    val sourceTopic = "flink_kafka_source"
    properties.setProperty("bootstrap.servers", bootStrap)
    properties.setProperty("group.id", "demo")
    val kafkaSource = new FlinkKafkaConsumer[String](sourceTopic, new SimpleStringSchema(), properties)
    val stream = env
      .addSource(kafkaSource)
    stream.flatMap(_.split(","))
        .map((_,1))
        .keyBy(_._1)
        .sum(1)
        .print()
    env.execute(this.getClass.getSimpleName)
  }

}

• 运行结果
• 生产者

>spark
>spark,hadoop

• IDEA 控制台

1> (spark,1)
11> (hadoop,1)
1> (spark,2)

3. 使用 Kafka 作为 Sink

3.1 Code

package com.xk.bigdata.flink.datastream.connector

import java.util.Properties

import org.apache.flink.api.common.serialization.SimpleStringSchema
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.scala._
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer

/**
 * 数据输出到 Kafka
 */
object KafkaSinkApp {

  def main(args: Array[String]): Unit = {
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    val stream = env.readTextFile("data/wc.txt")
    val bootStrap = "bigdatatest02:9092,bigdatatest03:9092,bigdatatest04:9092"
    val topic = "flink_kafka_sink"

    val myProducer = new FlinkKafkaProducer[String](
      bootStrap, // target topic
      topic,
      new SimpleStringSchema()) // serialization schema

    stream.addSink(myProducer)
    env.execute(this.getClass.getSimpleName)
  }

}

• Kafka Consumer

spark
hadoop,spark,flink
spark,hadoop