Spark 之 WholeStageCodegen

CodeGen framework

• CodegenSupport(接口)
  相邻Operator通过Produce-Consume模式生成代码。
  Produce生成整体处理的框架代码，例如aggregation生成的代码框架如下:

if (!initialized) {
 # create a hash map, then build the aggregation hash map
 # call child.produce()
 initialized = true;
}
while (hashmap.hasNext()) {
 row = hashmap.next();
 # build the aggregation results
 # create variables for results
 # call consume(), which will call parent.doConsume()
 if (shouldStop()) return;
}

Consume生成当前节点处理上游输入的Row的逻辑。如Filter生成代码如下:

# code to evaluate the predicate expression, result is isNull1 and value2
if (!isNull1 && value2) {
 # call consume(), which will call parent.doConsume()
}