32.DataStream API之Connectors(Fault Tolerance Guarantees)

最新推荐文章于 2022-06-13 09:52:36 发布

li2327234939

最新推荐文章于 2022-06-13 09:52:36 发布

阅读量135

点赞数

分类专栏： Flink官方文档笔记文章标签： Fault Tolerance Guarantees Connectors DataStream API Apache flink

版权

45 篇文章 5 订阅

订阅专栏

flink 1.9

Flink的容错机制能够确保在出现故障时，恢复程序并继续执行它们。这些故障包括机器硬件故障、网络故障、短暂的程序故障等。

只有当source源参与了快照机制时，Flink才能确保仅一次exactly-once状态更新到用户定义的状态。下表列出了flink与绑定的连接器的状态更新保证模式。

Source	Guarantees	Notes
Apache Kafka	exactly once	Use the appropriate Kafka connector for your version https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/connectors/kafka.html
AWS Kinesis Streams	exactly once	https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/connectors/kinesis.html
RabbitMQ	at most once (v 0.10) / exactly once (v 1.0)	https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/connectors/rabbitmq.html
Twitter Streaming API	at most once	https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/connectors/twitter.html
Google PubSub	at least once	https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/connectors/pubsub.html
Collections	exactly once
Files	exactly once
Sockets	at most once

具体的细节可以参考各种连接器相关内容。

为了确保端到端end-to-end的仅一次exactly-once记录发送(除了支持exactly-once状态语义之外)，数据接收器sink需要参与检查点机制。下表列出了Flink与内置接收器sink的发送担保模式(假设只进行一次状态更新):

关注