https://beam.apache.org/get-started/beam-overview/
Apache Beam Overview
Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, you build a program that defines the pipeline. The pipeline is then executed by one of Beam’s supported distributed processing back-ends, which include Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.
Beam is particularly useful for Embarrassingly Parallel data processing tasks, in which the problem can be decomposed into many smaller bundles of data that can be processed independently and in parallel. You can also use Beam for Extract, Transform, and Load (ETL) tasks and pure data integration. These tasks are useful for moving data between different storage media and data sources, transforming data into a more desirable format, or loading data onto a new system.
Apache Beam SDKs
The Beam SDKs provide a unified programming model that can represent and transform data sets of any size, whether the input is a finite data set from a batch data source, or an infinite data set from a streaming data source. The Beam SDKs use the same classes to represent both bounded and unbounded data, and the same transforms to operate on that data. You use the Beam SDK of your choice to build a program that defines your data processing pipeline.
Apache Beam Pipeline Runners
The Beam Pipeline Runners translate the data processing pipeline you define with your Beam program into the API compatible with the distributed processing back-end of your choice. When you run your Beam program, you’ll need to specify an appropriate runner for the back-end where you want to execute your pipeline.
Beam currently supports Runners that work with the following distributed processing back-ends:
- Apache Apex
- Apache Flink
- Apache Apache Gearpump (incubating)
- Apache Spark
- Google Cloud Dataflow
Note: You can always execute your pipeline locally for testing and debugging purposes.
Apache Beam 是一种开源统一模型,用于定义批处理和流处理的数据并行处理管道。使用 Beam SDK 可以构建定义管道的程序,然后由支持的分布式处理后端执行,包括 Apache Flink、Apache Spark 和 Google Cloud Dataflow。Beam 特别适用于可以分解为许多独立并行处理的小数据包的尴尬并行任务。
1万+

被折叠的 条评论
为什么被折叠?



