APACHE BEAM2.0.0支持的是SPARK1.6版本,如果要支持SPARK2.0.2版本需要做部分改造
改造内容一:
废弃原有的Accumulator接口,实现 AccumulatorV2
需要实现
public class NamedAccumulatorV2 extends AccumulatorV2<NamedAggregators, NamedAggregators>
public class MetricsAccumlatorV2 extends AccumulatorV2<MetricsContainerStepMap, MetricsContainerStepMap>并更改代码中相关引用的内容改造内容二:SparkRunner批处理模式中增加执行器退出的代码
startPipeline = executorService.submit( new Runnable() { @Override public void run() { pipeline.traverseTopologically(new Evaluator(translator, evaluationContext)); evaluationContext.computeOutputs(); LOG.info("Batch pipeline execution complete."); executorService.shutdown(); } });