SparkStreaming对接Kafka0.10+管理offset到zookeeper的方式

版本号：
spark-streaming-kafka-0-10_2.11
  version:2.4.0
kafka-clients
  version:0.11.0.0

问题：之前都是使用的0.8版本的来保存offset，但因为线上Kafka集群版本为0.11.0.0的，保存offset的方式发生了很大的变化。

官网的方式：

import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.kafka.common.serialization.StringDeserializer
import org.apache.spark.streaming.kafka010._
import org.apache.spark.streaming.kafka010.LocationStrategies.PreferConsistent
import org.apache.spark.streaming.kafka010.ConsumerStrategies.Subscribe

val kafkaParams = Map[String, Object](
  "bootstrap.servers" -> "localhost:9092,anotherhost:9092",
  "key.deserializer" -> classOf[StringDeserializer],
  "value.deserializer" -> classOf[StringDeserializer],
  "group.id" -> "use_a_separate_group_id_for_each_stream",
  "auto.offset.reset" -> "latest",
  "enable.auto.commit" -> (false: java.lang.Boolean)
)

val topics = Array("topicA", "topicB")
val stream = KafkaUtils.createDirectStream[String, String](
  streamingContext,
  PreferConsistent,
  Subscribe[String, String](topics, kafkaParams)
)

stream.foreachRDD { rdd =>
  val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
  rdd.foreachPartition { iter =>
    val o: OffsetRange = offsetRanges(TaskContext.get.partitionId)
    println(s"${o.topic} ${o.partition} ${o.fromOffset} ${o.untilOffset}")
  }
  // some time later, after outputs have completed
  stream.asInstanceOf[CanCommitOffsets].commitAsync(offsetRanges)
}

 而我现在有个需求就是在foreachRDD之前做一下开窗函数和map、filter运算，但做完这些操作之后再进行foreachRDD就获取不到offsetRanges了。改好之后的代码：

//全局的
var offsetRanges: Array[OffsetRange] = Array[OffsetRange]()

//方法体内的
val kafkaParams = Map[String, Object](
    "bootstrap.servers" -> PropertiesUtil.getPropertiesToStr("kafka.hosts"),
    "key.deserializer" -> classOf[StringDeserializer],
    "value.deserializer" -> classOf[StringDeserializer],
    "group.id" -> "xxx",
    "auto.offset.reset" -> "latest",
    "enable.auto.commit" -> (false: java.lang.Boolean)
)

val topics = Array("topicA")

val messages: InputDStream[ConsumerRecord[String, String]] = KafkaOffsetUtil.createMyZookeeperDirectKafkaStream(
    ssc,
    kafkaParams,
    topics,
    PropertiesUtil.getPropertiesToStr("zookeeper.group.name"),
    PropertiesUtil.getPropertiesToStr("zookeeper.path"))

val dataOriginDStream = messages.transform {
    rdd =>
        offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges
        rdd
}.filter(kv => {
    ...
})

dataOriginDStream.foreachRDD(rdd => {
    if (!rdd.isEmpty()) {
        ...
    }
    // 存储新的offset
    KafkaOffsetUtil.saveOffsets(
        PropertiesUtil.getPropertiesToStr("zookeeper.path"),
        offsetRanges,
        PropertiesUtil.getPropertiesToStr("zookeeper.group.name"))
})

 我的思路：使用transform先将offsetRanges保存到一个变量中，然后foreachRDD的时候再取出使用！！