Spark-Streaming中得Kafka数据源|Kafka 0-8 Receive模式|Kafka 0-8 Direct模式|Kafka 0-10 Direct模式

版本选型

1. ReceiverAPI：需要一个专门的Executor去接收数据，然后发送给其他的Executor做计算。存在的问题，接收数据的Executor和计算的Executor速度会有所不同，特别在接收数据的Executor速度大于计算的Executor速度，会导致计算数据的节点内存溢出。
2. DirectAPI：是由计算的Executor来主动消费Kafka的数据，速度由自身控制。（高版本后，不支持Receiver了）

Kafka 0-8 Receive模式

1. 导入依赖

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
    <version>2.1.1</version>
</dependency>

2. 代码实现

package com.atguigu.kafka

import org.apache.spark.SparkConf
import org.apache.spark.streaming.dstream.{
   DStream, ReceiverInputDStream}
import org.apache.spark.streaming.kafka.KafkaUtils
import org.apache.spark.streaming.{
   Seconds, StreamingContext}

object Receiver_Test {
   
  def main(args: Array[String]): Unit = {
   
    //1.创建SparkConf
    val sparkConf: SparkConf = new SparkConf().setAppName("ReceiverWordCount").setMaster("local[*]")

    //2.创建StreamingContext
    val ssc = new StreamingContext(sparkConf, Seconds(3))

    // 使用receiverAPI 读取kafka数据 创建DStrem
    val kafkaDStrem: ReceiverInputDStream[(String, String)] = KafkaUtils.createStream(ssc,
      "hadopp102:2181,hadopp102:2181,hadopp102:2181",
      "big",
      Map("test" -> 2))


    // 计算wordcount 并打印
    val lineDStream: DStream[String] = kafkaDStrem.map(_._2)
    lineDStream.flatMap(_.split(" ")).map((_,1)).reduceByKey(_ + _).print()

    //5.开启任务
    ssc.start()
    ssc