spark源码bug fix: Block broadcast_465 not found

前言

使用spark

一、spark源码bug fix: Block broadcast_465 not found

示例：pandas 是基于NumPy 的一种工具，该工具是为了解决数据分析任务而创建的。

二、问题描述

21/08/25 20:26:18 ERROR spark.TaskContextImpl: Error in TaskCompletionListener
java.lang.IllegalStateException: Block broadcast_465 not found
	at org.apache.spark.storage.BlockInfoManager$$anonfun$1.apply(BlockInfoManager.scala:288)
	at org.apache.spark.storage.BlockInfoManager$$anonfun$1.apply(BlockInfoManager.scala:288)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.storage.BlockInfoManager.unlock(BlockInfoManager.scala:287)
	at org.apache.spark.storage.BlockManager.releaseLock(BlockManager.scala:657)
	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$releaseLock$1.apply(TorrentBroadcast.scala:246)
	at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$releaseLock$1.apply(TorrentBroadcast.scala:246)
	at org.apache.spark.TaskContext$$anon$1.onTaskCompletion(TaskContext.scala:123)
	at org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:97)
	at org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:95)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:95)
	at org.apache.spark.scheduler.Task.run(Task.scala:112)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

2.问题分析

代码如下（示例）：

BlockInfoManager.scala unlock方法源码

  /**
   * Release a lock on the given block.
   */
  def unlock(blockId: BlockId): Unit = synchronized {
    logTrace(s"Task $currentTaskAttemptId releasing lock for $blockId")
    val info = get(blockId).getOrElse {
      throw new IllegalStateException(s"Block $blockId not found")
    }
    if (info.writerTask != BlockInfo.NO_WRITER) {
      info.writerTask = BlockInfo.NO_WRITER
      writeLocksByTask.removeBinding(currentTaskAttemptId, blockId)
    } else {
      assert(info.readerCount > 0, s"Block $blockId is not locked for reading")
      info.readerCount -= 1
      val countsForTask = readLocksByTask(currentTaskAttemptId)
      val newPinCountForTask: Int = countsForTask.remove(blockId, 1) - 1
      assert(newPinCountForTask >= 0,
        s"Task $currentTaskAttemptId release lock on block $blockId more times than it acquired it")
    }
    notifyAll()
  }

val info = get(blockId).getOrElse {
throw new IllegalStateException(s"Block $blockId not found")
}
get不到就抛异常，改为return就OK, 编译源码，打包

为什么会发生这种情况：删除block的命令重复发，造成，可能是同个节点或者不同节点的任务发送过来

总结

提示：这里对文章进行总结：
看到问题不用怕，分析一下，fix就完事