Spark Streaming官方文档翻译基本概念之累加器、广播变量和检查点

Spark Streaming官方文档翻译Spark Streaming总览
Spark Streaming官方文档翻译基本概念之初始化与Dstream
Spark Streaming官方文档翻译基本概念之输入数据流和接收器
Spark Streaming官方文档翻译基本概念之转换操作
Spark Streaming官方文档翻译基本概念之输出操作
Spark Streaming官方文档翻译基本概念之sql与Mllib
Spark Streaming官方文档基本概念之缓存与检查点
Spark Streaming官方文档翻译基本概念之累加器、广播变量和检查点
Spark Streaming官方文档翻译Spark Streaming应用之部署，升级，监控
Spark Streaming官方文档翻译Spark Streaming性能调优
Spark Streaming官方文档翻译Spark Streaming容错
Spark Streaming官方文档翻译Spark Streaming +Kafka 集成指南
Spark Streaming官方文档翻译Spark Streaming自定义接收器

基本概念

累加器、广播变量和检查点(Accumulators, Broadcast Variables, and Checkpoints)

无法从Spark Streaming中的检查点恢复累加器和广播变量。如果启用了检查点并同时使用累加器或广播变量，则必须为累加器和广播变量创建延迟实例化的单例实例，以便在驱动程序失败重新启动后重新实例化它们。如下面的例子所示。

object WordBlacklist {

  @volatile private var instance: Broadcast[Seq[String]] = null

  def getInstance(sc: SparkContext): Broadcast[Seq[String]] = {
    if (instance == null) {
      synchronized {
        if (instance == null) {
          val wordBlacklist = Seq("a", "b", "c")
          instance = sc.broadcast(wordBlacklist)
        }
      }
    }
    instance
  }
}

object DroppedWordsCounter {

  @volatile private var instance: LongAccumulator = null

  def getInstance(sc: SparkContext): LongAccumulator = {
    if (instance == null) {
      synchronized {
        if (instance == null) {
          instance = sc.longAccumulator("WordsInBlacklistCounter")
        }
      }
    }
    instance
  }
}

wordCounts.foreachRDD { (rdd: RDD[(String, Int)], time: Time) =>
  // Get or register the blacklist Broadcast
  val blacklist = WordBlacklist.getInstance(rdd.sparkContext)
  // Get or register the droppedWordsCounter Accumulator
  val droppedWordsCounter = DroppedWordsCounter.getInstance(rdd.sparkContext)
  // Use blacklist to drop words and use droppedWordsCounter to count them
  val counts = rdd.filter { case (word, count) =>
    if (blacklist.value.contains(word)) {
      droppedWordsCounter.add(count)
      false
    } else {
      true
    }
  }.collect().mkString("[", ", ", "]")
  val output = "Counts at time " + time + " " + counts
})

查看完整的源代码。