Spark Streaming源码解读之State管理之updateStateByKey和mapWithState解密

最新推荐文章于 2023-11-26 12:14:00 发布

snail_gesture

最新推荐文章于 2023-11-26 12:14:00 发布

阅读量6.3k

点赞数 4

分类专栏： Spark Streaming源码详解文章标签： spark hadoop java scala 大数据

本文链接：https://blog.csdn.net/snail_gesture/article/details/51510588

版权

本文介绍了Spark Streaming中用于状态管理的关键方法updateStateByKey和mapWithState，分析了它们在处理大数据时如何维护状态。updateStateByKey在PairDStreamFunctions中实现，基于历史数据更新；mapWithState则返回MapWithStateDStream，允许更灵活的状态维护。由于涉及历史数据处理，当数据量大时，updateStateByKey可能效率不高。

摘要由CSDN通过智能技术生成

背景：
整个Spark Streaming是按照Batch Duractions划分Job的。但是很多时候我们需要算过去的一天甚至一周的数据，这个时候不可避免的要进行状态管理，而Spark Streaming每个Batch Duractions都会产生一个Job，Job里面都是RDD，所以此时面临的问题就是怎么对状态进行维护？这个时候就需要借助updateStateByKey和mapWithState方法完成核心的步骤。
源码分析：
1. 无论是updateStateByKey还是mapWithState方法在DStream中均没有，但是是通过隐身转换函数实现其功能。

object DStream {

  // `toPairDStreamFunctions` was in SparkContext before 1.3 and users had to
  // `import StreamingContext._` to enable it. Now we move it here to make the compiler find
  // it automatically. However, we still keep the old function in StreamingContext for backward
  // compatibility and forward to the following function directly.

  implicit def toPairDStreamFunctions[K, V](stream: DStream[(K, V)])
      (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null):
    PairDStreamFunctions[K, V] = {
   
    new PairDStreamFunctions[K, V](stream)
  }

updateStateByKey：
1. 在PairDStreamFunctions中updateStateByKey具体实现如下：
在已有的历史基础上，updateFunc对历史数据进行更新。该函数的返回值是DStream类型的。

/**
 * Return a new "state" DStream where the state for each key is updated by applying
 * the given function on the previous state of the key and the new values of each key.
 * Hash partitioning is used to generate the RDDs with Spark's default number of partitions.
 * @param updateFunc State update function. If `this` function returns None, then
 *                   corresponding state key-value pair will be eliminated.
 * @tparam S State type
 */
def updateStateByKey[S: ClassTag](
    updateFunc: (Seq[V], Option[S]) => Option[S]
  ): DStream[(K, S)] = ssc.withScope {
// defaultPartitioner
  updateStateByKey(updateFunc, defaultPartitioner())
}

2.  defaultPartitioner:

private[streaming] def defaultPartitioner(numPartitions: Int = self.ssc.sc.defaultParallelism) = {
  new HashPartitioner(numPartitions)
}

3.  partitioner就是控制RDD的每个patition

/**
 * Return a new "state" DStream where the state for each key is updated by applying
 * the given function on the previous state of the key and the new values of the key.
 * org.apache.spark.Partitioner is used to control the partitioning of each RDD.
 * @param updateFunc State update function. If `this` function returns None, then
 *                   corresponding state key-value pair will be eliminated.
 * @param partitioner Partitioner for controlling the partitioning of each RDD in the new
 *                    DStream.
 * @tparam S State type
 */
def updateStateByKey[S: ClassTag](
    updateFunc: (Seq[V], Option[S]) => Option[S],
    partitioner: Partitioner
  ): DStream[(K, S)] = ssc.withScope {
  val cleanedUpdateF = sparkContext.clean(updateFunc)
  val newUpdateFunc = (iterator: Iterator[(K, Seq[V], Option[S])]) => {
    iterator.flatMap(t => cleanedUpdateF(t._2, t._3).map(s => (t._1, s)))
  }
  updateStateByKey(newUpdateFunc, partitioner, true)
}

4.  rememberPartitioner默认为true

/**
 * Return a new "state" DStream where the state for each key is updated by applying
 * the given function on the previous state of the key and the new values of each key.
 * org.apache.spark.Partitioner