flink的mysql两阶段提交_Flink 源码之两阶段提交

Flink 系列博客

Flink QuickStart

Flink双流操作

Flink on Yarn Kerberos的配置

Flink on Yarn部署和任务提交操作

Flink配置Prometheus监控

Flink in docker 部署

Flink HA 部署

Flink 常见调优参数总结

Flink 源码之任务提交流程分析

Flink 源码之基本算子

Flink 源码之Trigger

Flink 源码之Evictor

Flink 源码之Window

Flink 源码之WindowOperator

Flink 源码之StreamGraph生成

Flink 源码之JobGraph生成

两阶段提交协议

两阶段提交协议针对Flink的Sink。要求下游的系统支持事务，或者是幂等性。两阶段提交是指如下两个阶段：

preCommit： 预提交。在Sink进行snapshot操作的时候调用此方法。

commit： 真正的提交操作。当系统中各个operator的checkpoint操作都成功之后，JobManager会通知各个operator checkpoint操作已完成。此时会调用该方法。

TwoPhaseCommitSinkFunction

该类是实现两阶段提交Sink的父类，封装了两阶段提交的主要逻辑。

initializeState方法。该方法在CheckpointedFunction接口中定义，在集群中执行的时候调用，用于初始化状态后端。

该方法主要有以下逻辑：

获取状态存储变量state。

提交所有已经执行过preCommit的事务。

终止所有尚未preCommit的事务。

创建一个新事务。

代码如下：

@Override

public void initializeState(FunctionInitializationContext context) throws Exception {

// when we are restoring state with pendingCommitTransactions, we don't really know whether the

// transactions were already committed, or whether there was a failure between

// completing the checkpoint on the master, and notifying the writer here.

// (the common case is actually that is was already committed, the window

// between the commit on the master and the notification here is very small)

// it is possible to not have any transactions at all if there was a failure before

// the first completed checkpoint, or in case of a scale-out event, where some of the

// new task do not have and transactions assigned to check)

// we can have more than one transaction to check in case of a scale-in event, or

// for the reasons discussed in the 'notifyCheckpointComplete()' method.

// 获取状态存储

state = context.getOperatorStateStore().getListState(stateDescriptor);

boolean recoveredUserContext = false;

// 从上一个snapshot恢复完成的时候返回true，如果任务不支持snapshot，永远返回false

if (context.isRestored()) {

LOG.info("{} - restoring state", name());

for (State operatorState : state.get()) {

userContext = operatorState.getContext();

// 获取待提交的事务

// 在snapshotState方法调用preCommit之后，事务会被存储到该列表

List> recoveredTransactions = operatorState.getPendingCommitTransactions();

List handledTransactions = new ArrayList<>(recoveredTransactions.size() + 1);

for (TransactionHolder recoveredTransaction : recoveredTransactions) {

// If this fails to succeed eventually, there is actually data loss

// 恢复并提交这些之前在state中保存的事务

recoverAndCommitInternal(recoveredTransaction);

handledTransactions.add(recoveredTransaction.handle);

LOG.info("{} committed recovered trans