Flink CDC 3.0 Starrocks建表失败会导致任务卡主!

Flink CDC 3.0 Starrocks建表失败会导致任务卡主!

现象

StarRocks建表失败,然后任务自动重启,重启完毕后数据回放,jobMaster打印下面日志后,整个任务会卡主

There are already processing requests. Wait for processing

原因分析

前提概要:可以先阅读CDC表变更处理流程然后再读下面会更加清晰

涉及类包括SchemaRegistrySchemaOperatorStarRocksMetadataApplier

SchemaRegistry->handleEventFromOperator方法执行建表失败后会导致任务重启,但是jobMaster不会重启,因此SchemaRegistry.requestHandler.pendingSchemaChanges无法删除导致任务卡主!

public void flushSuccess(TableId tableId, int sinkSubtask) {
    flushedSinkWriters.add(sinkSubtask);
    if (flushedSinkWriters.equals(activeSinkWriters)) {
        LOG.info(
                "All sink subtask have flushed for table {}. Start to apply schema change.",
                tableId.toString());
        PendingSchemaChange waitFlushSuccess = pendingSchemaChanges.get(0);
        //执行表结构变更操作!
        applySchemaChange(tableId, waitFlushSuccess.getChangeRequest().getSchemaChangeEvent());
        waitFlushSuccess.getResponseFuture().complete(wrap(new ReleaseUpstreamResponse()));

        if (RECEIVED_RELEASE_REQUEST.equals(waitFlushSuccess.getStatus())) {
            //异常会跳过删除pendingSchame!
            startNextSchemaChangeRequest();
        }
    }
}
//删除pendingSchemaChanges中已经完成的pendingSchame
private void startNextSchemaChangeRequest() {
    this.pendingSchemaChanges.remove(0);
    this.flushedSinkWriters.clear();
    ...
}

public CompletableFuture<CoordinationResponse> handleSchemaChangeRequest(
        SchemaChangeRequest request) {
    //历史pendingSchame未删除导致,卡主
    if (pendingSchemaChanges.isEmpty()) {
        LOG.info(
                "Received schema change event request from table {}. Start to buffer requests for others.",
                request.getTableId().toString());
        if (request.getSchemaChangeEvent() instanceof CreateTableEvent
                && schemaManager.schemaExists(request.getTableId())) {
            return CompletableFuture.completedFuture(wrap(new SchemaChangeResponse(false)));
        }
        CompletableFuture<CoordinationResponse> response =
                CompletableFuture.completedFuture(wrap(new SchemaChangeResponse(true)));
        schemaManager.applySchemaChange(request.getSchemaChangeEvent());
        pendingSchemaChanges.add(new PendingSchemaChange(request, response));
        pendingSchemaChanges.get(0).startToWaitForReleaseRequest();
        return response;
    } else {
        LOG.info("There are already processing requests. Wait for processing.");
        CompletableFuture<CoordinationResponse> response = new CompletableFuture<>();
        pendingSchemaChanges.add(new PendingSchemaChange(request, response));
        return response;
    }
}

解决办法

  1. 让建表执行成功
  2. catch住异常,将schame删除后再异常重启,代码如下所示
  3. 在Starrocks抛出异常时,将异常类转换为添加有注解@ThrowableAnnotation(ThrowableType.NonRecoverableError)的异常类
  4. 官方已经解决:https://github.com/apache/flink-cdc/pull/3128 (SchemaOperator算子状态初始化时,发送sendRequestToCoordinator(new RefreshPendingListsRequest())清空SchemaRegistryRequestHandlerpendingSchemaChanges
//com.ververica.cdc.runtime.operators.schema.coordinator.SchemaRegistryRequestHandler类
public void flushSuccess(TableId tableId, int sinkSubtask) {
    flushedSinkWriters.add(sinkSubtask);
    if (flushedSinkWriters.equals(activeSinkWriters)) {
        LOG.info(
                "All sink subtask have flushed for table {}. Start to apply schema change.",
                tableId.toString());
        PendingSchemaChange waitFlushSuccess = pendingSchemaChanges.get(0);
        try {
            applySchemaChange(
                    tableId, waitFlushSuccess.getChangeRequest().getSchemaChangeEvent());
        } catch (Exception e) {
//           2 解决办法  catch住所有的exception可能不严谨
//           pendingSchemaChanges.remove(0);
//           flushedSinkWriters.clear();
            //3解决办法 这里还可以考虑在实现端控制抛出可恢复还是不可恢复类型的异常!例如在StarRocksMetadataApplier的applyCreateTable方法中将RuntimeException 改为添加有ThrowableAnnotation注解的异常类!
            throw new MyException(e);
        }
        waitFlushSuccess.getResponseFuture().complete(wrap(new ReleaseUpstreamResponse()));

        if (RECEIVED_RELEASE_REQUEST.equals(waitFlushSuccess.getStatus())) {
            startNextSchemaChangeRequest();
        }
    }

@ThrowableAnnotation(ThrowableType.NonRecoverableError)
private static class MyException extends RuntimeException {

    public MyException(Throwable cause) {
        super(cause);
    }
}
  • 10
    点赞
  • 6
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
关于flink-cdc建表操作,我们可以参考官方文档提供的示例代码。在官方文档的"Table Sink"部分,有一个示例展示了如何使用flink-cdc创建一个MySQL表。首先,你需要创建一个TableEnvironment对象,然后使用该对象创建一个Catalog,并将该Catalog注册到TableEnvironment中。接下来,你可以使用TableEnvironment的executeSql方法执行创建表的SQL语句。下面是一个示例代码: ```java import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.EnvironmentSettings; import org.apache.flink.table.api.TableEnvironment; public class FlinkCDCExample { public static void main(String[] args) { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); EnvironmentSettings settings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build(); TableEnvironment tEnv = TableEnvironment.create(settings); // 创建一个MySQL的Catalog tEnv.executeSql("CREATE CATALOG mycatalog WITH (\n" + " 'type'='jdbc',\n" + " 'property-version'='1',\n" + " 'base-url'='jdbc:mysql://localhost:3306',\n" + " 'default-database'='mydatabase',\n" + " 'username'='myuser',\n" + " 'password'='mypassword',\n" + " 'jdbc-driver'='com.mysql.cj.jdbc.Driver'\n" + ")"); // 注册Catalog到TableEnvironment tEnv.useCatalog("mycatalog"); // 创建MySQL表 tEnv.executeSql("CREATE TABLE mytable (\n" + " id INT,\n" + " name STRING\n" + ") WITH (\n" + " 'connector'='mysql-cdc',\n" + " 'database-name'='mydatabase',\n" + " 'table-name'='mytable',\n" + " 'debezium.deployment.mode'='local'\n" + ")"); } } ``` 以上代码创建了一个名为`mytable`的MySQL表,表结构包含`id`和`name`两个字段。你可以根据自己的需求修改表名和字段。同时,根据实际情况修改MySQL的连接信息,包括URL、用户名和密码等。请确保你已经将相应的依赖项添加到项目的依赖文件中,并导入所需的类库。 需要注意的是,以上示例使用的是MySQL作为示例数据库,如果你使用的是其他数据库,你需要相应地修改连接信息和驱动程序。 希望以上信息对你有所帮助。如有任何疑问,请随时提问。<span class="em">1</span><span class="em">2</span><span class="em">3</span> #### 引用[.reference_title] - *1* *2* *3* [浅谈flink-cdc的使用](https://blog.csdn.net/chenzuoli/article/details/122021562)[target="_blank" data-report-click={"spm":"1018.2226.3001.9630","extra":{"utm_source":"vip_chatgpt_common_search_pc_result","utm_medium":"distribute.pc_search_result.none-task-cask-2~all~insert_cask~default-1-null.142^v93^chatsearchT3_1"}}] [.reference_item style="max-width: 100%"] [ .reference_list ]

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值