16.Spark Streaming源码解读之数据清理机制解析

最新推荐文章于 2022-03-09 10:45:50 发布

听风的蜗牛

最新推荐文章于 2022-03-09 10:45:50 发布

阅读量2.6k

点赞数 2

分类专栏： spark streaming 进阶文章标签： spark spark Streaming 源码解析 Job DStream

本文链接：https://blog.csdn.net/zhouzx2010/article/details/51801262

版权

spark streaming 进阶专栏收录该内容

11 篇文章 0 订阅

订阅专栏

原创文章，转载请注明:转载自听风居士博客(http://blog.csdn.net/zhouzx2010)

本期内容：

一、Spark Streaming 数据清理总览

二、Spark Streaming 数据清理过程详解

三、Spark Streaming 数据清理的触发机制

Spark Streaming不像普通Spark 的应用程序，普通Spark程序运行完成后,中间数据会随着SparkContext的关闭而被销毁,而Spark Streaming一直在运行，不断计算，每一秒中在不断运行都会产生大量的中间数据，所以需要对对象及元数据需要定期清理。每个batch duration运行时不断触发job后需要清理rdd和元数据。下面我们就结合源码详细解析一下Spark Streaming程序的数据清理机制。

一、数据清理总览

Spark Streaming 运行过程中,随着时间不断产生Job,当job运行结束后,需要清理相应的数据(RDD,元数据信息,Checkpoint数据),Job由JobGenerator定时产生,数据的清理也是有JobGenerator负责。

JobGenerator负责数据清理控制的代码位于一个消息循环体eventLoop中：

  
  
   
   
    
        eventLoop = new EventLoop[JobGeneratorEvent]("JobGenerator") {
   
   
   
   
    
          override protected def onReceive(event: JobGeneratorEvent): Unit = processEvent(event)
   
   
   
   
    
     
   
   
   
   
    
          override protected def onError(e: Throwable): Unit = {
   
   
   
   
    
            jobScheduler.reportError("Error in job generator", e)
   
   
   
   
    
          }
   
   
   
   
    
        }
   
   
   
   
    
        eventLoop.start()

 
 其中的核心逻辑位于processEvent(event 
 )函数中: 

 
       /** Processes all events */
    
       private def processEvent(event: JobGeneratorEvent) {
    
         logDebug("Got event " + event)
    
         event match {
    
           case GenerateJobs(time) => generateJobs(time)
    
           case ClearMetadata(time) => clearMetadata(time)
    
           case DoCheckpoint(time, clearCheckpointDataLater) =>
    
             doCheckpoint(time, clearCheckpointDataLater)
    
           case ClearCheckpointData(time) => clearCheckpointData(time)
    
         }
    
       }

 
 可以看到当JobGenerator收到ClearMetadata(time) 和 ClearCheckpointData(time)是会进行相应的数据清理,其中  
 clearMetadata 
 ( 
 time 
 ) 
 会清理RDD数据和一些元数据信息, C 
 learCheckpointData 
 ( 
 time 
 ) 
 会清理Checkpoint数据。 

 
 二、数据清理过程详解 

 
     2.1 ClearMetaData 过程详解

 
 首先看一下clearMetaData函数的处理逻辑： 

 
       /** Clear DStream metadata for the given `time`. */
    
       private def clearMetadata(time: Time) {
    
         ssc.graph.clearMetadata(time)
    
         // If checkpointing is enabled, then checkpoint,
    
         // else mark batch to be fully processed
    
         if (shouldCheckpoint) {
    
           eventLoop.post(DoCheckpoint(time, clearCheckpointDataLater = true))
    
         } else {
    
           // If checkpointing is not enabled, then delete metadata information about
    
           // received blocks (block data not saved in any case). Otherwise, wait for
    
           // checkpointing of this batch to complete.
    
           val maxRememberDuration = graph.getMaxInputStreamRememberDuration()
    
           jobScheduler.receiverTracker.cleanupOldBlocksAndBatches(time - maxRememberDuration)
    
           jobScheduler.inputInfoTracker.cleanup(time - maxRememberDuration)
    
           markBatchFullyProcessed(time)
    
         }
    
       }

 
 首先调用了DStreamGraph的clearMetadata方法: 

 
       def clearMetadata(time: Time) {
    
         logDebug("Clearing metadata for time " + time)
    
         this.synchronized {
    
           outputStreams.foreach(_.clearMetadata(time))
    
         }
    
         logDebug("Cleared old metadata for time " + time)
    
       }

 
 这里调用了所有OutputDStream (关于DStream 的分类请参考 
 http://blog.csdn.net/zhouzx2010/article/details/51460790 
 )的clearMetadata方法 

 
       private[streaming] def clearMetadata(time: Time) {
    
         val unpersistData = ssc.conf.getBoolean("spark.streaming.unpersist", true)
    
          //获取需要清理的RDD
    
         val oldRDDs = generatedRDDs.filter(_._1 <= (time - rememberDuration))
    
         logDebug("Clearing references to old RDDs: [" +
    
           oldRDDs.map(x => s"${x._1} -> ${x._2.id}").mkString(", ") + "]")
    
         //将要清除的RDD从generatedRDDs 中清除 
    
         generatedRDDs --= oldRDDs.keys
    
         if (unpersistData) {
    
           logDebug(s"Unpersisting old RDDs: ${oldRDDs.values.map(_.id).mkString(", ")}")
    
           oldRDDs.values.foreach { rdd =>
    
         //将RDD 从persistence列表中移除
    
             rdd.unpersist(false)
    
             // Explicitly remove blocks of BlockRDD
    
             rdd match {
    
               case b: BlockRDD[_] =>
    
                 logInfo(s"Removing blocks of RDD $b of time $time")
    
                 //移除RDD的block 数据
    
                 b.removeBlocks()
    
               case _ =>
    
             }
    
           }
    
         }
    
         logDebug(s"Cleared ${oldRDDs.size} RDDs that were older than " +
    
           s"${time - rememberDuration}: ${oldRDDs.keys.mkString(", ")}")
    
         //清除依赖的DStream
    
         dependencies.foreach(_.clearMetadata(time))
    
       }

 
 关键的清理逻辑在代码中做了详细注释,首先清理DStream对应的RDD的元数据信息,然后清理RDD的数据,最后对DStream所依赖的DStream进行清理。 

 
 回到JobGenerator的clearMetadata函数： 

 
         if (shouldCheckpoint) {
    
           eventLoop.post(DoCheckpoint(time, clearCheckpointDataLater = true))
    
         } else {
    
           // If checkpointing is not enabled, then delete metadata information about
    
           // received blocks (block data not saved in any case). Otherwise, wait for
    
           // checkpointing of this batch to complete.
    
           val maxRememberDuration = graph.getMaxInputStreamRememberDuration()
    
           jobScheduler.receiverTracker.cleanupOldBlocksAndBatches(time - maxRememberDuration)
    
           jobScheduler.inputInfoTracker.cleanup(time - maxRememberDuration)
    
           markBatchFullyProcessed(time)
    
         }

 
 调用了ReceiverTracker的  
 cleanupOldBlocksAndBatches 
 方法，最后调用了clearupOldBatches方法： 

 
       def cleanupOldBatches(cleanupThreshTime: Time, waitForCompletion: Boolean): Unit = synchronized {
    
         require(cleanupThreshTime.milliseconds < clock.getTimeMillis())
    
         val timesToCleanup = timeToAllocatedBlocks.keys.filter { _ < cleanupThreshTime }.toSeq
    
         logInfo(s"Deleting batches: ${timesToCleanup.mkString(" ")}")
    
         if (writeToLog(BatchCleanupEvent(timesToCleanup))) {
    
           //将要删除的Batch数据清除
    
           timeToAllocatedBlocks --= timesToCleanup
    
           //清理WAL日志
    
           writeAheadLogOption.foreach(_.clean(cleanupThreshTime.milliseconds, waitForCompletion))
    
         } else {
    
           logWarning("Failed to acknowledge batch clean up in the Write Ahead Log.")
    
         }
    
       }

  可以看到 
 ReceiverTracker的 
 clearupOldBatches方法 
 清理了Receiver数据,也就是Batch数据和WAL日志数据。 

  最后对InputInfoTracker信息进行清理： 

  
  
   
   
    
      def cleanup(batchThreshTime: Time): Unit = synchronized {
   
   
   
   
    
        val timesToCleanup = batchTimeToInputInfos.keys.filter(_ < batchThreshTime)
   
   
   
   
    
        logInfo(s"remove old batch metadata: ${timesToCleanup.mkString(" ")}")
   
   
   
   
    
        batchTimeToInputInfos --= timesToCleanup
   
   
   
   
    
      }

  这简单的清除了 
 batchTimeToInputInfos  
 的输入信息。 

 
 2.2 ClearCheckPoint 过程详解

 
 看一下clearCheckpointData的处理逻辑:

 
       /** Clear DStream checkpoint data for the given `time`. */
    
       private def clearCheckpointData(time: Time) {
    
         ssc.graph.clearCheckpointData(time)
    
         // All the checkpoint information about which batches have been processed, etc have
    
         // been saved to checkpoints, so its safe to delete block metadata and data WAL files
    
         val maxRememberDuration = graph.getMaxInputStreamRememberDuration()
    
         jobScheduler.receiverTracker.cleanupOldBlocksAndBatches(time - maxRememberDuration)
    
         jobScheduler.inputInfoTracker.cleanup(time - maxRememberDuration)
    
         markBatchFullyProcessed(time)
    
       }
    
  后面的ReceiverTraker和InputInforTracker的清理逻辑和ClearMetaData的相同,这分析DStreamGraph的clearCheckpointData方法:

 
       def clearCheckpointData(time: Time) {
    
         logInfo("Clearing checkpoint data for time " + time)
    
         this.synchronized {
    
           outputStreams.foreach(_.clearCheckpointData(time))
    
         }
    
         logInfo("Cleared checkpoint data for time " + time)
    
       }

 
 同样的调用了 
 DStreamGraph中所有OutputDStream的clearCheckPiontData 方法: 

 
       private[streaming] def clearCheckpointData(time: Time) {
    
         logDebug("Clearing checkpoint data")
    
         checkpointData.cleanup(time)
    
         dependencies.foreach(_.clearCheckpointData(time))
    
         logDebug("Cleared checkpoint data")
    
       }

 
 这里的核心逻辑在 
 checkpointData 
 .cleanup(time) 
 方法,这里的CheckpointData 是 DStreamCheckpointData对象, 
  DStreamCheckpointData的clearup方法如下: 

 
     def cleanup(time: Time) {
    
         // 获取需要清理的Checkpoint 文件 时间
    
         timeToOldestCheckpointFileTime.remove(time) match {
    
           case Some(lastCheckpointFileTime) =>
    
             //获取需要删除的文件
    
             val filesToDelete = timeToCheckpointFile.filter(_._1 < lastCheckpointFileTime)
    
             logDebug("Files to delete:\n" + filesToDelete.mkString(","))
    
             filesToDelete.foreach {
    
               case (time, file) =>
    
                 try {
    
                   val path = new Path(file)
    
                   if (fileSystem == null) {
    
                     fileSystem = path.getFileSystem(dstream.ssc.sparkContext.hadoopConfiguration)
    
                   }
    
               //
     
       删除文件  
     
                   fileSystem.delete(path, true)
    
                   timeToCheckpointFile -= time
    
                   logInfo("Deleted checkpoint file '" + file + "' for time " + time)
    
                 } catch {
    
                   case e: Exception =>
    
                     logWarning("Error deleting old checkpoint file '" + file + "' for time " + time, e)
    
                     fileSystem = null
    
                 }
    
             }
    
           case None =>
    
             logDebug("Nothing to delete")
    
         }
    
       }
    
   可以看到checkpoint的清理,就是删除了指定时间以前的checkpoint文件。

 
 三、数据清理的触发

 
     3 
 .1 ClearMetaData 过程的触发

 
 JobGenerator 生成job后,交给JobHandler执行, 
  JobHandler的run方法中,会在job执行完后给JobScheduler 发送JobCompleted消息: 

  
  
   
   
    
              _eventLoop = eventLoop
   
   
   
   
    
              if (_eventLoop != null) {
   
   
   
   
    
                _eventLoop.post(JobCompleted(job, clock.getTimeMillis()))
   
   
   
   
    
     
   
   
   
   
    
              }

  JobScheduler 收到 
 JobCompleted  
 消息调用  
 handleJobCompletion  
 方法,源码如下: 

 
       private def processEvent(event: JobSchedulerEvent) {
    
         try {
    
           event match {
    
             case JobStarted(job, startTime) => handleJobStart(job, startTime)
    
             case JobCompleted(job, completedTime) => handleJobCompletion(job, completedTime)
    
             case ErrorReported(m, e) => handleError(m, e)
    
           }
    
         } catch {
    
           case e: Throwable =>
    
             reportError("Error in job scheduler", e)
    
         }
    
       }

 
 在 JobScheduler 的handleJobCompletion方法中会调用JobGenerator的 
 onBatchCompletion 
 方法,我们看一下 
 JobGenerator的  
 onBatchCompletion  
 方法的源码: 

 
       def onBatchCompletion(time: Time) {
    
         eventLoop.post(ClearMetadata(time))
    
       }

  可以看到 
 JobGenerator的 
 onBatchCompletion 
 方法给自己发送了ClearMetadata消息从而触发了ClearMetaData操作。 

 
 3.2 ClearCheckPoint 过程的触发 

 
     清理CheckPoint数据发生在CheckPoint完成之后,我们先看一下CheckPointHandler的run方法:

 
               // All done, print success
    
               val finishTime = System.currentTimeMillis()
    
               logInfo("Checkpoint for time " + checkpointTime + " saved to file '" + checkpointFile +
    
                 "', took " + bytes.length + " bytes and " + (finishTime - startTime) + " ms")
    
               //调用JobGenerator的方法进行checkpoint数据清理
    
               jobGenerator.onCheckpointCompletion(checkpointTime, clearCheckpointDataLater)
    
               return

 
 可以看到在checkpoint完成后,会调用JobGenerator的onCheckpointCompletion方法进行checkpoint数据清理,我查看 
 JobGenerator的onCheckpointCompletion方法源码: 

 
       def onCheckpointCompletion(time: Time, clearCheckpointDataLater: Boolean) {
    
         if (clearCheckpointDataLater) {
    
           eventLoop.post(ClearCheckpointData(time))
    
         }
    
       }

 
 可以看到 
 JobGenerator的onCheckpointCompletion方法中首先对传进来的  
 clearCheckpointDataLater  
 参数进行判断,如果该参数为true,就会给JobGenerator的eventLoop循环体发送ClearCheckpointData消息,从而触发 
 clearCheckpointData 
  方法的调用,进行Checkpoint数据的清理。 

 
 什么时候该参数会true呢？ 

 
 我们回到JobGenerator的  
 ClearMetadata 
  方法： 

 
       private def clearMetadata(time: Time) {
    
         ssc.graph.clearMetadata(time)
    
         if (shouldCheckpoint) {
    
           //发送DoCheckpoint消息,并进行相应的Checkpoint数据清理
    
           eventLoop.post(DoCheckpoint(time, clearCheckpointDataLater = true))
    
         } else {
    
           val maxRememberDuration = graph.getMaxInputStreamRememberDuration()
    
           jobScheduler.receiverTracker.cleanupOldBlocksAndBatches(time - maxRememberDuration)
    
           jobScheduler.inputInfoTracker.cleanup(time - maxRememberDuration)
    
           markBatchFullyProcessed(time)
    
         }
    
       }

  可以看到在clearMetadata方法中,发送了DoCheckpoint消息,其中参数  
 clearCheckpointDataLater 
  为ture。Generator的eventLoop收到该消息后调用  
 doCheckpoint 
  方法： 

 
       private def doCheckpoint(time: Time, clearCheckpointDataLater: Boolean) {
    
         if (shouldCheckpoint && (time - graph.zeroTime).isMultipleOf(ssc.checkpointDuration)) {
    
           logInfo("Checkpointing graph for time " + time)
    
           ssc.graph.updateCheckpointData(time)
    
           checkpointWriter.write(new Checkpoint(ssc, time), clearCheckpointDataLater)
    
         }
    
       }

  这里关键一步：调用了CheckpointWriter的write方法,注意此时参数  
 clearCheckpointDataLater 
  为true。我们进入该方法： 

 
       def write(checkpoint: Checkpoint, clearCheckpointDataLater: Boolean) {
    
         try {
    
           val bytes = Checkpoint.serialize(checkpoint, conf)
    
          //将参数clearCheckpointDataLater传入CheckpoitWriteHandler
    
           executor.execute(new CheckpointWriteHandler(
    
             checkpoint.checkpointTime, bytes, clearCheckpointDataLater))
    
           logInfo("Submitted checkpoint of time " + checkpoint.checkpointTime + " writer queue")
    
         } catch {
    
           case rej: RejectedExecutionException =>
    
             logError("Could not submit checkpoint task to the thread pool executor", rej)
    
         }
    
       }

 
 可以看到此时参数  
 clearCheckpointDataLater 
   
 传入 
 CheckpointWriteHandler 
  。这样Checkpoint完成之后就会发送 
 ClearCheckpointData 
 消息给JobGenerator进行Checkpoint数据的清理。 

 
  
 原创文章，转载请注明: 
 转载自 听风居士博客( 
 http://blog.csdn.net/zhouzx2010) 

听风的蜗牛

关注

2
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
16.Spark Streaming源码解读之数据清理机制解析

原创文章，转载请注明:转载自听风居士博客(http://blog.csdn.net/zhouzx2010)本期内容：一、Spark Streaming 数据清理总览二、Spark Streaming 数据清理过程详解三、Spark Streaming 数据清理的触发机制 Spark Streaming不像普通Spark 的应用程序，普
复制链接

扫一扫