Beam学习笔记（3）：Flink Streaming Pipeline Translator

最新推荐文章于 2022-10-26 14:56:33 发布

Aegeaner

最新推荐文章于 2022-10-26 14:56:33 发布

阅读量767

点赞数

本文链接：https://blog.csdn.net/Aegeaner/article/details/70768016

版权

FlinkStreamingPipelineTranslator

1 所有的可用Translator：

// here you can find all the available translators.
  static {
    TRANSLATORS.put(Read.Bounded.class, new BoundedReadSourceTranslator());
    TRANSLATORS.put(Read.Unbounded.class, new UnboundedReadSourceTranslator());
    TRANSLATORS.put(TextIO.Write.Bound.class, new TextIOWriteBoundStreamingTranslator());

    TRANSLATORS.put(ParDo.MultiOutput.class, new ParDoStreamingTranslator());
    TRANSLATORS.put(
        SplittableParDo.ProcessElements.class, new SplittableProcessElementsStreamingTranslator());
    TRANSLATORS.put(
        SplittableParDo.GBKIntoKeyedWorkItems.class, new GBKIntoKeyedWorkItemsTranslator());


    TRANSLATORS.put(Window.Assign.class, new WindowAssignTranslator());
    TRANSLATORS.put(Flatten.PCollections.class, new FlattenPCollectionTranslator());
    TRANSLATORS.put(
        FlinkStreamingViewOverrides.CreateFlinkPCollectionView.class,
        new CreateViewStreamingTranslator());

    TRANSLATORS.put(Reshuffle.class, new ReshuffleTranslatorStreaming());
    TRANSLATORS.put(GroupByKey.class, new GroupByKeyTranslator());
    TRANSLATORS.put(Combine.PerKey.class, new CombinePerKeyTranslator());
  }

2 applyStreamingTransform：

基本逻辑：将PTransform关联到streamingContext

// create the applied PTransform on the streamingContext
    streamingContext.setCurrentTransform(node.toAppliedPTransform());
    typedTranslator.translateNode(typedTransform, streamingContext);

12种类型的translator多态实现了StreamTransformTranslator.translateNode(T transform, FlinkStreamingTranslationContext context);方法。