spark stream 3.0.0 scala版本接收kafka消息并计算wordcount

环境准备需要kafka，kafka又依赖于zookeeper。

kafka命令行操作包括启动kafka，创建topic，启动消息生产者等操作，windows环境下的具体命令可以参考在Windows安装部署和测试Kafka收发消息

这里选择的是scala的2.12版本sdk，本地安装的也是2.12的安装包，所以选择spark-streaming-kafka-0-10_2.12。

在使用spark-streaming-kafka-0-8_2.11时，连接kafka的方法时参数会有所不同。

1. 添加依赖

<dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
            <version>3.0.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.12</artifactId>
            <version>3.0.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-10_2.12</artifactId>
            <version>3.1.0</version>
        </dependency>
</dependencies>

2. 编写测试代码

package com.demo

import org.apache.kafka.common.serialization.StringDeserializer
import org.apache.spark.SparkConf
import org.apache.spark.streaming.kafka010.{ConsumerStrategies, KafkaUtils, LocationStrategies}
import org.apache.spark.streaming.{Seconds, StreamingContext}


object SparkStreamKafkaDirectAPI {

  val getSSC1: () => StreamingContext = () => {
    val sparkConf: SparkConf = new
        SparkConf().setAppName("ReceiverWordCount").setMaster("local[*]")
    val ssc = new StreamingContext(sparkConf, Seconds(3))
    ssc
  }

  def getSSC: StreamingContext = {
    //1.创建 SparkConf
    val sparkConf: SparkConf = new
        SparkConf().setAppName("ReceiverWordCount").setMaster("local[*]")
    //2.创建 StreamingContext
    val ssc = new StreamingContext(sparkConf, Seconds(3))
    //设置 CK
    ssc.checkpoint("./ck2")
    //3.定义 Kafka 参数
    val brokers = "192.168.22.56:9092"

    val topics = Array("spark-topic")
    val kafkaParams = Map[String, Object](
      "bootstrap.servers" -> brokers,
      "key.deserializer" -> classOf[StringDeserializer],
      "value.deserializer" -> classOf[StringDeserializer],
      "group.id" -> "gid",
      "auto.offset.reset" -> "latest",
      "enable.auto.commit" -> (false: java.lang.Boolean)
    )

    //4.读取 Kafka 数据
    val kafkaDStream = KafkaUtils.createDirectStream[String, String](ssc,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe[String, String](topics, kafkaParams)
    )

    kafkaDStream.print();

    val lines = kafkaDStream.map(x => {
      x.value()
    })

    //5.计算 WordCount
    lines.flatMap(_.split(" "))
      .map((_, 1))
      .reduceByKey(_ + _)
      .print()

    //6.返回数据
    ssc
  }

    def main(args: Array[String]): Unit = {
      //获取 SSC
      val ssc: StreamingContext = StreamingContext.getActiveOrCreate("./ck2", () =>
        getSSC)

      ssc.sparkContext.setLogLevel("WARN")

      //开启任务
      ssc.start()
      ssc.awaitTermination()
    }

}

3. 执行测试

kafka命令行消息生产者输入：

>hello world
>hello spark
>hello stream

在idea控制台输出：

-------------------------------------------
Time: 1644997512000 ms
-------------------------------------------
(hello,1)
(world,1)

-------------------------------------------
Time: 1644997515000 ms
-------------------------------------------
(hello,1)
(spark,1)