Spark 源码理解之withScope (含补充说明)

Spark 源码理解之withScope

在通过看RDD源码理解各算子的作用时, 总能看到withScope, withScope到底是个什么东西?
首先需要了解几个东西: scala柯里化(currying), 贷出模式(loan pattern)

scala 柯里化(currying)

在scala 中，一个经过柯里化的函数在应用时支持多个参数列表，而不是只有一个。
当第一次调用只传入第一个参数时，返回一个用于第二次调用的函数值。

scala> def curriedSum(x: Int)(y: Int) = x + y
curriedSum: (x: Int)(y: Int)
scala> curriedSum(1)(2)
res1: Int = 3
scala> val add3 = curriedSum(3)_
add3: Int => Int = <function1>
scala> add3(4)
res2: Int = 7

curriedSum(3)_中_是占位符，表示第二个参数先不传，返回值是一个函数值。

我们看RDD 源码

/**
  * Execute a block of code in a scope such that all new RDDs created in this body will
  * be part of the same scope. For more detail, see {{org.apache.spark.rdd.RDDOperationScope}}.
  *
  * Note: Return statements are NOT allowed in the given body.
  */
  private[spark] def withScope[U](body: => U): U = RDDOperationScope.withScope[U](sc)(body) // 这里用了柯里化

贷出模式(loan pattern)

把公共部分(函数体)抽出来封装成方法，把非公共部分通过函数值传进来

/**
  * Execute the given body such that all RDDs created in this body will have the same scope.
  *
  * If nesting is allowed, any subsequent calls to this method in the given body will instantiate
  * child scopes that are nested within our scope. Otherwise, these calls will take no effect.
  *
  * Additionally, the caller of this method may optionally ignore the configurations and scopes
  * set by the higher level caller. In this case, this method will ignore the parent caller's
  * intention to disallow nesting, and the new scope instantiated will not have a parent. This
  * is useful for scoping physical operations in Spark SQL, for instance.
  *
  * Note: Return statements are NOT allowed in body.
  */
  private[spark] def withScope[T](
      sc: SparkContext,
      name: String,
      allowNesting: Boolean,
      ignoreParent: Boolean)(body: => T): T = {
    // Save the old scope to restore it later
    val scopeKey = SparkContext.RDD_SCOPE_KEY
    val noOverrideKey = SparkContext.RDD_SCOPE_NO_OVERRIDE_KEY
    val oldScopeJson = sc.getLocalProperty(scopeKey)
    val oldScope = Option(oldScopeJson).map(RDDOperationScope.fromJson)
    val oldNoOverride = sc.getLocalProperty(noOverrideKey)
    try {
      if (ignoreParent) {
        // Ignore all parent settings and scopes and start afresh with our own root scope
        sc.setLocalProperty(scopeKey, new RDDOperationScope(name).toJson)
      } else if (sc.getLocalProperty(noOverrideKey) == null) {
        // Otherwise, set the scope only if the higher level caller allows us to do so
        sc.setLocalProperty(scopeKey, new RDDOperationScope(name, oldScope).toJson)
      }
      // Optionally disallow the child body to override our scope
      if (!allowNesting) {
        sc.setLocalProperty(noOverrideKey, "true")
      }
      body // 非公共部分
    } finally {
      // Remember to restore any state that was modified before exiting
      sc.setLocalProperty(scopeKey, oldScopeJson)
      sc.setLocalProperty(noOverrideKey, oldNoOverride)
    }
  }

因为每个RDD算子方法，都有共同部分和共同参数，所以这里用withScope封装了公共部分代码，用柯里化把共同参数先传进去。
然后非公共部分代码，通过第二个参数传进去。这里body参数就是非公共部分函数值。

理解算子

/**
  * Return a new RDD by applying a function to all elements of this RDD.
  */
  def map[U: ClassTag](f: T => U): RDD[U] = withScope {
    val cleanF = sc.clean(f)
    new MapPartitionsRDD[U, T](this, (context, pid, iter) => iter.map(cleanF))
  }

这里就举一个例子, map 方法
其实这里是相当于直接调用withScope 方法，后面花括号里面的是函数字面量，是参数。
在scala中，当方法只有一个参数时，后面可以用花括号代替圆括号。

补充说明

之所以叫做withScope是因为正如上面所说的该Scope指的是公共部分
补充：withScope也是个函数名，名字可以随便起
它相当于前面curriedSum(x: Int)(y: Int)例子中的curriedSum(x: Int)这一部分
它把算子的函数签名中大家都相同的那部分给包裹了起来，剩下只需填写个性化的部分
抛开晦涩难懂的scala，来看个shell中的变量与函数的例子做个类比

#!/bin/bash

# 定义函数1
demo_123() {
  echo "Demo 123"
}

# 定义函数2
demo_abc() {
  echo "Demo abc"
}

echo "正常使用函数名调用函数"
demo_123
demo_abc

# 提取函数名的`公共部分`
WITHSCOPE="demo_"
echo "通过 withScope 调用函数"
$WITHSCOPE"123"
$WITHSCOPE"abc"
## WITHSCOPE 是公共部分，其后面是各个函数的个性化内容

本文已经解释得相当清楚了，仔细阅读就会明白

参考资料

Spark源码理解之 withScope