Kafka Source + Flink+ValueState+scala

kafka作为Flink的source，ValueState完成状态存储

flink代码

package com.jy.flink

import java.util.Properties

import org.apache.flink.api.common.functions.RichMapFunction
import org.apache.flink.api.common.serialization.SimpleStringSchema
import org.apache.flink.api.common.state.{ValueState, ValueStateDescriptor}
import org.apache.flink.configuration.Configuration
import org.apache.flink.streaming.api.scala._
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer
import org.apache.kafka.clients.consumer.ConsumerConfig

/**
  * @author t
  * @date 2020/4/16 14:38
  * @version 1.0
  * @company jy
  */
object FlinkAndValueState {
  def main(args: Array[String]): Unit = {
    val environment = StreamExecutionEnvironment.getExecutionEnvironment
    //kafka consumer连接参数
    val prop1 = new Properties()
    prop1.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "spark:9092")
    prop1.put(ConsumerConfig.GROUP_ID_CONFIG, "g1")
    //创建kafka sources、kafka sink
    val kafkaSource = new FlinkKafkaConsumer[String]("a", new SimpleStringSchema(), prop1)
    //数据转换
    environment
      .addSource(kafkaSource)
      .flatMap(_.split("\\s+"))
      .map((_,1))
      .keyBy(0)
      .map(new RichMapFunction[(String, Int), (String, Int)] {
        var valueState: ValueState[Int] = _

        override def open(parameters: Configuration): Unit = {
          val count = new ValueStateDescriptor[Int]("count", createTypeInformation[Int])
          valueState = getRuntimeContext.getState(count)
        }

        override def map(value: (String, Int)): (String, Int) = {
          var historyState = valueState.value()
          if (historyState == 0) {
            historyState = 0
          }
          valueState.update(historyState + value._2)
          (value._1, valueState.value())

        }
      })
      .print()
    environment.execute("state")

  }

}

结果输出

5> (s,1)
2> (kk,1)
2> (i,1)
2> (kk,2)
4> (ss,1)
5> (s,2)
2> (kk,3)
7> (oo,1)
2> (h,1)
4> (ss,2)
2> (kk,4)

kafka主题创建

[root@spark kafka_2.11-2.2.0]bin/kafka-topics.sh --bootstrap-server spark:9092 --topic a --partitions 1 --replication-factor 1 --create

kafka消息发送

[root@spark kafka_2.11-2.2.0]# bin/kafka-console-producer.sh --broker-list spark:9092 --topic a
>s a b
>s l
>kk
>s kk
>s kk
>i kk
>ss
>s
>kk
>oo
>h
>ss
>kk