Structured Streaming之StateStoreProvider

  这个类主要作用是状态更新，描述如下：

  An implementation of [[StateStoreProvider]] and [[StateStore]] in which all the data is backed by files in a HDFS-compatible file system. All updates to the store has to be done in sets transactionally, and each set of updates increments the store's version. These versions can be used to re-execute the updates (by retries in RDD operations) on the correct version of the store, and regenerate the store version.

     一个基于StateStoreProvider和StateStore的实现，所有的数据都由兼容HDFS系统的文件系统的文件支持。这些所有的更新都必须要以事务集合的方式完成而且每一组更新都会增加版本。这些版本可以用来重复执行更新（尝试RDD操作）在正确的版本上，并且再生版本。

* Fault-tolerance model:
* - Every set of updates is written to a delta file before committing.
* - The state store is responsible for managing, collapsing and cleaning up of delta files.
* - Multiple attempts to commit the same version of updates may overwrite each other.
*   Consistency guarantees depend on whether multiple attempts have the same updates and
*   the overwrite semantics of underlying file system.
* - Background maintenance of files ensures that last versions of the store is always recoverable
* to ensure re-executed RDD operations re-apply updates on the correct past version of the
* store.

来看看这个故障容错模型：

 1.所有的集合更新之前都会写出到一个delta文件。

2.这个StateStore代表着管理。折叠并且清除这个delta文件。

3.多样的尝试提交相同的版本更新可能会覆盖。一致性保证依赖于其他的多样的尝试是否有相同的更新和底层文件系统的覆盖语义。

4.文件后台维护的文件确保最后的版本总是可以维护的。确保可以执行正确的版本。

关注点：1.如何维护最后版本？这个设计是一组为粒度的有什么坏处呢？如果是你设计又会如何设计呢？

看看如何获取到版本文件的：

//基础目录
private val baseDir =
  new Path(id.checkpointLocation, s"${id.operatorId}/${id.partitionId.toString}")

private def fetchFiles(): Seq[StoreFile] = {
//拿到所有分区文件
  val files: Seq[FileStatus] = try {
    fs.listStatus(baseDir)
  } catch {
    case _: java.io.FileNotFoundException =>
      Seq.empty
  }
//定义一个类型为版本，文件的集合
  val versionToFiles = new mutable.HashMap[Long, StoreFile]
//一个文件一个文件遍量
  files.foreach { status =>
    val path = status.getPath
    val nameParts = path.getName.split("\\.")
//路径名称切分
    if (nameParts.size == 2) {
      val version = nameParts(0).toLong
      nameParts(1).toLowerCase match {
        case "delta" =>
          // ignore the file otherwise, snapshot file already exists for that batch id
          if (!versionToFiles.contains(version)) {
//如果是delta版本的就放入集合版本路径存储信息。
            versionToFiles.put(version, StoreFile(version, path, isSnapshot = false))
          }
        case "snapshot" =>
 //如果是个快照版本也放入集合
          versionToFiles.put(version, StoreFile(version, path, isSnapshot = true))
        case _ =>
          logWarning(s"Could not identify file $path for $this")
      }
    }
  }
//再进行一下排序，最后返回
  val storeFiles = versionToFiles.values.toSeq.sortBy(_.version)
  logDebug(s"Current set of files for $this: ${storeFiles.mkString(", ")}")
  storeFiles
}

设计的有什么问题呢？很明显如果状态过多会导致HashMap直接内存溢出基于内存的弊端吧。

俄罗斯小伙实现了一个方法https://docs.databricks.com/spark/latest/structured-streaming/production.html

自己实现也行，并不是特别复杂。

再看一下写快照文件的过程：

private def doSnapshot(): Unit = {
  try {
//获取文件
    val files = fetchFiles()
    if (files.nonEmpty) {
//获取最后一个版本
      val lastVersion = files.last.version
//寻找最后的delta版本
      val deltaFilesForLastVersion =
        filesForVersion(files, lastVersion).filter(_.isSnapshot == false)
//获取最大版本的数据信息
      synchronized { loadedMaps.get(lastVersion) } match {
        case Some(map) =>
//进行快照操作
          if (deltaFilesForLastVersion.size > storeConf.minDeltasForSnapshot) {
            writeSnapshotFile(lastVersion, map)
          }
        case None =>
          // The last map is not loaded, probably some other instance is in charge
      }

    }
  } catch {
    case NonFatal(e) =>
      logWarning(s"Error doing snapshots for $this", e)
  }
}