spark中updateStateByKey引发StackOverflowError的解决

spark中updateStateByKey引发StackOverflowError的解决

问题描述

写的spark程序, 运行几个小时候总是会出现 StackOverflowError.

程序使用 spark-1.1 运行.

代码的逻辑大概是:

streamB = streamA.map().fiter().recudeByKeyAndWindow()
streamC = streamB.updateStateByKey()
streamD = streamC.updateStateByKey()
streamD.foreachRDD()

原因

1. 逐行注释代码, 发现error由updateStateByKey引发.
2. 在每个 updateStateByKey 之后用 foreachRDD 输出 toDebugString, 发现是因为 rdd 的 dependency chain 太长导致的StackOverflowError.

这就比较奇怪, 因为 updateStateByKey 默认会 checkpoint, 而 checkpoint 会切断 dependency chain.

可参考: spark streaming checkpointing

默认的checkpoint机制和interval

spark在代码中有 updateStateByKey 和 reduceByKeyAndWindow设置有 inverse function 的时候, 会自动 checkpoint .

checkpoint 的 interval 必须是 duration 的整数倍, 默认值为满足下面条件的最小值:

interval >= max(duration, Duration(10000))
interval % duration = 0

其中 duration 是 streaming 的 batch duration.

如 spark streaming checkpointing 所讲

checkpointing too infrequently causes the lineage and task sizes to grow which may have detrimental effects

如果将 batch duration 设置为 1 ms, checkpoint 的 interval 则会默认设为 10000 ms, 此时 batch 相对于 interval 来说就是 too infrequently. 这种设置下updateStateByKey很快就会引发 StackOverflowError.

另外需要注意的是, 每个 job 只会 checkpoint 一次. 可看源码:

  /**
   * Run a function on a given set of partitions in an RDD and pass the results to the given
   * handler function. This is the main entry point for all actions in Spark. The allowLocal
   * flag specifies whether the scheduler can run the computation on the driver rather than
   * shipping it out to the cluster, for short actions like first().
   */
  def runJob[T, U: ClassTag](
      rdd: RDD[T],
      func: (TaskContext, Iterator[T]) => U,
      partitions: Seq[Int],
      allowLocal: Boolean,
      resultHandler: (Int, U) => Unit) {
    if (dagScheduler == null) {
      throw new SparkException("SparkContext has been shutdown")
    }
    val callSite = getCallSite
    val cleanedFunc = clean(func)
    logInfo("Starting job: " + callSite.shortForm)
    dagScheduler.runJob(rdd, cleanedFunc, partitions, callSite, allowLocal,
      resultHandler, localProperties.get)
    progressBar.foreach(_.finishAll())
    rdd.doCheckpoint()
  }

也就是说, 下面的代码只会 checkpoint 一次. (spark streaming中, 一个 output operation 有一个 job.)

streamC = streamB.updateStateByKey()
streamD = streamC.updateStateByKey()
streamD.foreachRDD()

streamC一直没有checkpoint, 从而导致 dependency chain 过长产生 StackOverflowError.

foreachRDD 和 print 的不同

我将代码改为:

streamB = streamA.map().fiter().recudeByKeyAndWindow()
streamC = streamB.updateStateByKey()
streamC.foreachRDD()
streamD = streamC.updateStateByKey()
streamD.foreachRDD()

发现虽然 streamC 的 checkpoint 已经生效, 但是它的 dependency chain 依旧没有截断. 个人陷入困境.

leader Ning 使用 print 而非 foreachRDD, 发现 dependency chain 有被截断.

foreachRDD 源码:

  /**
   * Apply a function to each RDD in this DStream. This is an output operator, so
   * 'this' DStream will be registered as an output stream and therefore materialized.
   */
  def foreachRDD(foreachFunc: (RDD[T], Time) => Unit) {
    // because the DStream is reachable from the outer object here, and because 
    // DStreams can't be serialized with closures, we can't proactively check 
    // it for serializability and so we pass the optional false to SparkContext.clean
    new ForEachDStream(this, context.sparkContext.clean(foreachFunc, false)).register()
  }

print 源码:

  /**
   * Print the first ten elements of each RDD generated in this DStream. This is an output
   * operator, so this DStream will be registered as an output stream and there materialized.
   */
  def print() {
    def foreachFunc = (rdd: RDD[T], time: Time) => {
      val first11 = rdd.take(11)
      println ("-------------------------------------------")
      println ("Time: " + time)
      println ("-------------------------------------------")
      first11.take(10).foreach(println)
      if (first11.size > 10) println("...")
      println()
    }
    new ForEachDStream(this, context.sparkContext.clean(foreachFunc)).register()
  }

发现两者都是调用了 ForEachDStream, 区别在于 foreachRDD 设置了 context.sparkContext.clean 的 checkSerializable 为 false.

sparkContext.clean 的源码为:

  /**
   * Clean a closure to make it ready to serialized and send to tasks
   * (removes unreferenced variables in $outer's, updates REPL variables)
   * If <tt>checkSerializable</tt> is set, <tt>clean</tt> will also proactively
   * check to see if <tt>f</tt> is serializable and throw a <tt>SparkException</tt>
   * if not.
   *
   * @param f the closure to clean
   * @param checkSerializable whether or not to immediately check <tt>f</tt> for serializability
   * @throws <tt>SparkException<tt> if <tt>checkSerializable</tt> is set but <tt>f</tt> is not
   *   serializable
   */
  private[spark] def clean[F <: AnyRef](f: F, checkSerializable: Boolean = true): F = {
    ClosureCleaner.clean(f, checkSerializable)
    f
  }

至于 checkSerializable 为 false 导致 rdd 的 dependency chain 不截断的原因, 貌似和 一个KCore算法引发的StackOverflow奇案 相同, 都是 $outer 导致, 这里没有深究.

fix的方法

因此, 将代码调整为:

streamB = streamA.map().fiter().recudeByKeyAndWindow()
streamC = streamB.updateStateByKey()
streamC.print()
streamD = streamC.updateStateByKey()
streamD.foreachRDD()

虽然 print 会将数据传回 driver, 会有一点影响性能, 但是可以接受.