RocketMQ源码分析之主从同步消费信息

最新推荐文章于 2023-05-06 21:37:41 发布
最新推荐文章于 2023-05-06 21:37:41 发布
阅读量456 收藏 1
点赞数
分类专栏： RocketMQ 文章标签： RocketMQ
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/ph3636/article/details/90485116
版权

因为从broker也需要提供拉取消息的功能，所以需要把主broker的topic信息，订阅组信息，消息消费偏移量信息复制过来，当broker启动后会执行定时程序，一分钟同步一次

this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

    @Override
    public void run() {
        try {
            BrokerController.this.slaveSynchronize.syncAll();
        } catch (Throwable e) {
            log.error("ScheduledTask syncAll slave exception", e);
        }
    }
}, 1000 * 10, 1000 * 60, TimeUnit.MILLISECONDS);

public void syncAll() {
    this.syncTopicConfig();
    this.syncConsumerOffset();
    this.syncDelayOffset();
    this.syncSubscriptionGroupConfig();
}

同步topic配置信息，当两边的版本号不一致则替换为主broker的


private void syncTopicConfig() {
    String masterAddrBak = this.masterAddr;
    if (masterAddrBak != null) {
        try {
            TopicConfigSerializeWrapper topicWrapper =
                this.brokerController.getBrokerOuterAPI().getAllTopicConfig(masterAddrBak);
            if (!this.brokerController.getTopicConfigManager().getDataVersion()
                .equals(topicWrapper.getDataVersion())) {

                this.brokerController.getTopicConfigManager().getDataVersion()
                    .assignNewOne(topicWrapper.getDataVersion());
                this.brokerController.getTopicConfigManager().getTopicConfigTable().clear();
                this.brokerController.getTopicConfigManager().getTopicConfigTable()
                    .putAll(topicWrapper.getTopicConfigTable());
                this.brokerController.getTopicConfigManager().persist();

                log.info("Update slave topic config from master, {}", masterAddrBak);
            }
        } catch (Exception e) {
            log.error("SyncTopicConfig Exception, {}", masterAddrBak, e);
        }
    }
}

请求码为GET_ALL_TOPIC_CONFIG = 21


public TopicConfigSerializeWrapper getAllTopicConfig(
    final String addr) throws RemotingConnectException, RemotingSendRequestException,
    RemotingTimeoutException, InterruptedException, MQBrokerException {
    RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_TOPIC_CONFIG, null);

    RemotingCommand response = this.remotingClient.invokeSync(MixAll.brokerVIPChannel(true, addr), request, 3000);
    assert response != null;
    switch (response.getCode()) {
        case ResponseCode.SUCCESS: {
            return TopicConfigSerializeWrapper.decode(response.getBody(), TopicConfigSerializeWrapper.class);
        }
        default:
            break;
    }

    throw new MQBrokerException(response.getCode(), response.getRemark());
}

主broker打包信息返回


private RemotingCommand getAllTopicConfig(ChannelHandlerContext ctx, RemotingCommand request) {
    final RemotingCommand response = RemotingCommand.createResponseCommand(GetAllTopicConfigResponseHeader.class);
    // final GetAllTopicConfigResponseHeader responseHeader =
    // (GetAllTopicConfigResponseHeader) response.readCustomHeader();

    String content = this.brokerController.getTopicConfigManager().encode();
    if (content != null && content.length() > 0) {
        try {
            response.setBody(content.getBytes(MixAll.DEFAULT_CHARSET));
        } catch (UnsupportedEncodingException e) {
            log.error("", e);

            response.setCode(ResponseCode.SYSTEM_ERROR);
            response.setRemark("UnsupportedEncodingException " + e);
            return response;
        }
    } else {
        log.error("No topic in this broker, client: {}", ctx.channel().remoteAddress());
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark("No topic in this broker");
        return response;
    }

    response.setCode(ResponseCode.SUCCESS);
    response.setRemark(null);

    return response;
}

同步消费偏移量


private void syncConsumerOffset() {
    String masterAddrBak = this.masterAddr;
    if (masterAddrBak != null) {
        try {
            ConsumerOffsetSerializeWrapper offsetWrapper =
                this.brokerController.getBrokerOuterAPI().getAllConsumerOffset(masterAddrBak);
            this.brokerController.getConsumerOffsetManager().getOffsetTable()
                .putAll(offsetWrapper.getOffsetTable());
            this.brokerController.getConsumerOffsetManager().persist();
            log.info("Update slave consumer offset from master, {}", masterAddrBak);
        } catch (Exception e) {
            log.error("SyncConsumerOffset Exception, {}", masterAddrBak, e);
        }
    }
}

请求码为GET_ALL_CONSUMER_OFFSET = 43


public ConsumerOffsetSerializeWrapper getAllConsumerOffset(
    final String addr) throws InterruptedException, RemotingTimeoutException,
    RemotingSendRequestException, RemotingConnectException, MQBrokerException {
    RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_CONSUMER_OFFSET, null);
    RemotingCommand response = this.remotingClient.invokeSync(addr, request, 3000);
    assert response != null;
    switch (response.getCode()) {
        case ResponseCode.SUCCESS: {
            return ConsumerOffsetSerializeWrapper.decode(response.getBody(), ConsumerOffsetSerializeWrapper.class);
        }
        default:
            break;
    }

    throw new MQBrokerException(response.getCode(), response.getRemark());
}

打包返回存储并持久化

private RemotingCommand getAllConsumerOffset(ChannelHandlerContext ctx, RemotingCommand request) {
    final RemotingCommand response = RemotingCommand.createResponseCommand(null);

    String content = this.brokerController.getConsumerOffsetManager().encode();
    if (content != null && content.length() > 0) {
        try {
            response.setBody(content.getBytes(MixAll.DEFAULT_CHARSET));
        } catch (UnsupportedEncodingException e) {
            log.error("get all consumer offset from master error.", e);

            response.setCode(ResponseCode.SYSTEM_ERROR);
            response.setRemark("UnsupportedEncodingException " + e);
            return response;
        }
    } else {
        log.error("No consumer offset in this broker, client: {} ", ctx.channel().remoteAddress());
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark("No consumer offset in this broker");
        return response;
    }

    response.setCode(ResponseCode.SUCCESS);
    response.setRemark(null);

    return response;
}

同步延迟消息信息

private void syncDelayOffset() {
    String masterAddrBak = this.masterAddr;
    if (masterAddrBak != null) {
        try {
            String delayOffset =
                this.brokerController.getBrokerOuterAPI().getAllDelayOffset(masterAddrBak);
            if (delayOffset != null) {

                String fileName =
                    StorePathConfigHelper.getDelayOffsetStorePath(this.brokerController
                        .getMessageStoreConfig().getStorePathRootDir());
                try {
                    MixAll.string2File(delayOffset, fileName);
                } catch (IOException e) {
                    log.error("Persist file Exception, {}", fileName, e);
                }
            }
            log.info("Update slave delay offset from master, {}", masterAddrBak);
        } catch (Exception e) {
            log.error("SyncDelayOffset Exception, {}", masterAddrBak, e);
        }
    }
}

请求码为GET_ALL_DELAY_OFFSET = 45


public String getAllDelayOffset(
    final String addr) throws InterruptedException, RemotingTimeoutException, RemotingSendRequestException,
    RemotingConnectException, MQBrokerException, UnsupportedEncodingException {
    RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_DELAY_OFFSET, null);
    RemotingCommand response = this.remotingClient.invokeSync(addr, request, 3000);
    assert response != null;
    switch (response.getCode()) {
        case ResponseCode.SUCCESS: {
            return new String(response.getBody(), MixAll.DEFAULT_CHARSET);
        }
        default:
            break;
    }

    throw new MQBrokerException(response.getCode(), response.getRemark());
}

打包返回直接持久化

private RemotingCommand getAllDelayOffset(ChannelHandlerContext ctx, RemotingCommand request) {
    final RemotingCommand response = RemotingCommand.createResponseCommand(null);

    if (!(this.brokerController.getMessageStore() instanceof DefaultMessageStore)) {
        log.error("Delay offset not supported in this messagetore, client: {} ", ctx.channel().remoteAddress());
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark("Delay offset not supported in this messagetore");
        return response;
    }

    String content = ((DefaultMessageStore) this.brokerController.getMessageStore()).getScheduleMessageService().encode();
    if (content != null && content.length() > 0) {
        try {
            response.setBody(content.getBytes(MixAll.DEFAULT_CHARSET));
        } catch (UnsupportedEncodingException e) {
            log.error("Get all delay offset from master error.", e);

            response.setCode(ResponseCode.SYSTEM_ERROR);
            response.setRemark("UnsupportedEncodingException " + e);
            return response;
        }
    } else {
        log.error("No delay offset in this broker, client: {} ", ctx.channel().remoteAddress());
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark("No delay offset in this broker");
        return response;
    }

    response.setCode(ResponseCode.SUCCESS);
    response.setRemark(null);

    return response;
}

同步订阅组配置信息

private void syncSubscriptionGroupConfig() {
    String masterAddrBak = this.masterAddr;
    if (masterAddrBak != null) {
        try {
            SubscriptionGroupWrapper subscriptionWrapper =
                this.brokerController.getBrokerOuterAPI()
                    .getAllSubscriptionGroupConfig(masterAddrBak);

            if (!this.brokerController.getSubscriptionGroupManager().getDataVersion()
                .equals(subscriptionWrapper.getDataVersion())) {
                SubscriptionGroupManager subscriptionGroupManager =
                    this.brokerController.getSubscriptionGroupManager();
                subscriptionGroupManager.getDataVersion().assignNewOne(
                    subscriptionWrapper.getDataVersion());
                subscriptionGroupManager.getSubscriptionGroupTable().clear();
                subscriptionGroupManager.getSubscriptionGroupTable().putAll(
                    subscriptionWrapper.getSubscriptionGroupTable());
                subscriptionGroupManager.persist();
                log.info("Update slave Subscription Group from master, {}", masterAddrBak);
            }
        } catch (Exception e) {
            log.error("SyncSubscriptionGroup Exception, {}", masterAddrBak, e);
        }
    }
}

请求码为GET_ALL_SUBSCRIPTIONGROUP_CONFIG = 201

public SubscriptionGroupWrapper getAllSubscriptionGroupConfig(
    final String addr) throws InterruptedException, RemotingTimeoutException,
    RemotingSendRequestException, RemotingConnectException, MQBrokerException {
    RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.GET_ALL_SUBSCRIPTIONGROUP_CONFIG, null);
    RemotingCommand response = this.remotingClient.invokeSync(addr, request, 3000);
    assert response != null;
    switch (response.getCode()) {
        case ResponseCode.SUCCESS: {
            return SubscriptionGroupWrapper.decode(response.getBody(), SubscriptionGroupWrapper.class);
        }
        default:
            break;
    }

    throw new MQBrokerException(response.getCode(), response.getRemark());
}

打包返回存储并持久化

private RemotingCommand getAllSubscriptionGroup(ChannelHandlerContext ctx,
    RemotingCommand request) throws RemotingCommandException {
    final RemotingCommand response = RemotingCommand.createResponseCommand(null);
    String content = this.brokerController.getSubscriptionGroupManager().encode();
    if (content != null && content.length() > 0) {
        try {
            response.setBody(content.getBytes(MixAll.DEFAULT_CHARSET));
        } catch (UnsupportedEncodingException e) {
            log.error("", e);

            response.setCode(ResponseCode.SYSTEM_ERROR);
            response.setRemark("UnsupportedEncodingException " + e);
            return response;
        }
    } else {
        log.error("No subscription group in this broker, client:{} ", ctx.channel().remoteAddress());
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark("No subscription group in this broker");
        return response;
    }

    response.setCode(ResponseCode.SUCCESS);
    response.setRemark(null);

    return response;
}

 

ph3636
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Mysql主从同步的实现原理
09-09
主要介绍了Mysql主从同步的实现原理，小编觉得挺不错的，现在分享给大家，也给大家做个参考。一起跟随小编过来看看吧
实现VisualSVN Server主从同步备份.pdf
03-15
svn主从备份方便两个主机或者一个本地和远程之间备份提交的版本数据，实现双备份数据不易丢失。
redis主从同步介绍.pptx
最新发布
09-05
技术分享：redis主从同步介绍
RocketMQ——RocketMQ搭建及问题解决
SK_Jaco的博客
07-30 2377
功能：引用解耦、流量消峰、消息分发、保证最终一致性、方便动态扩容 下载 下载地址 RocketMQ 4.5.1 系统要求 64位 Linux、unix或mac; JDK版本1.8以上; 安装unzip yum install -y unzip 使用unzip命令解压 unzip rocketmq-all-4.5.1-bin-release.zip 单机 启动nameserver 两种启动方式...
【RocketMQ】【源码】主从模式下的消费进度管理
S-H_A-N
04-13 1121
消费者在启动的时候，会创建消息拉取API对象`PullAPIWrapper`，调用pullKernelImpl方法向Broker发送拉取消息的请求，那么在主从模式下消费者是如何选择向哪个Broker发送拉取请求的？
broker同步源码
wh710107079的博客
12-14 293
元数据
RocketMQ——Consumer篇：向Broker同步消费进度的定时任务
xiaomin1991222的专栏
10-13 362
每隔5秒调用一次MQClientInstance.persistAllConsumerOffset()方法将消费进度向Broker同步。遍历MQClientInstance.consumerTable: ConcurrentHashMap<String/*group */, MQConsumerInner>变量。对于PushConsumer端和PullConsumer端，处理逻辑是...
RocketMQ（六）多Master多Slave模式-异步复制集群搭建
weixiaohuai的博客
03-22 5470
目录 一、概述 二、环境准备 三、总体架构图 四、配置RocketMQ-Master01(主1 - 10.0.90.59) 五、配置RocketMQ-Master02(主2 - 10.0.90.144) 六、配置RocketMQ-Slave01(从1 - 10.0.91.8) 七、配置RocketMQ-Slave02(从2 - 10.0.91.49) 八、启动NameServer集群 九、启动Broker集群 十、启动可视化控制台 一、概述 前面一篇文章我们介绍了RocketMQ..
RocketMQ的主从同步
wangkai6666的博客
02-07 973
RocketMQ学习之主从同步
【RocketMQ】学习RocketMQ必须要知道的主从同步原理
浩瀚宇宙的一粒尘埃
04-21 3131
​ [TOC] RocketMQ的Broker分为Master和Slave两个角色，为了保证高可用性(HA)， Master角色的机器接收到消息后，要把内容同步到Slave机器上，这样一旦Master宕机，Slave机器依然可以提供服务。 下面介绍Master和Slave角色机器间同步功能实现的源码。 同步元数据 ​ Slave需要和Master同步的不只是消息本身，一些元数据信息也需要同步，比如TopicConfig信息、ConsumerOffset信息、DelayOffset和Subscription
Mysql主从同步备份策略分享
09-11
主从服务器上的MySQL数据库版本同为5.1.34
Shell脚本实战之DNS主从同步脚本实例
09-14
主要介绍了Shell脚本实战之DNS主从同步脚本实例，文中通过示例代码介绍的非常详细，对大家的学习或者工作具有一定的参考学习价值，需要的朋友们下面随着小编来一起学习学习吧
RocketMQ关于Broker闪断故障排查【实战笔记】
gaoliang1719的专栏
03-21 4471
目录一、问题描述 二、Broker日志分析 1.查看GC日志 2.查看Broker日志 三、系统监控分析 1.网络监控 2.磁盘IO监控 3.CP...
RocketMQ源码解析（四）——HA机制之主从同步
wukuruaCL的博客
05-12 2022
参考： 【RocketMQ】学习RocketMQ必须要知道的主从同步原理_Mr.ZhuRunHua的博客-CSDN博客_rocketmq主从同步原理 RocketMq 高可用实现逻辑_风火1989的博客-CSDN博客 rocketmq的broker如何同步信息的？ - notlate - 博客园 (cnblogs.com) 源码版本：RocketMQ 4.9.3 文章目录主从同步流程提出问题一、Slave 定时同步 Master 元数据二、Slave 同步 Master 消息数据HA 相关类介绍实例化.
RocketMQ安装以及源码启动
xhl1123456789的博客
05-06 925
Linux安装RocketMQ、RocketMQ源码下载调试。
rocketmq 之namesrv（十八）mqclient admin请求处理获取所有的订阅组配置信息
游语
03-03 413
获取所有的订阅组配置信息AdminBrokerProcessor#getAllSubscriptionGroup AdminBrokerProcessor#processRequest#this.getAllSubscriptionGroup(ctx, request) private RemotingCommand getAllSubscriptionGroup(ChannelHandlerCo...
RocketMQ（七） RocketMQ的两种消费模式
m0_67402970的博客
06-12 1304
目录一、概述二、集群消费三、广播消费RocketMQ主要提供了两种消费模式：集群消费以及广播消费。我们只需要在定义消费者的时候通过**setMessageModel(MessageModel.XXX)**方法就可以指定是集群还是广播式消费，默认是集群消费模式，即每个Consumer Group中的Consumer均摊所有的消息。下面我们通过简单的示例演示一下。一个 Consumer Group 中的 Consumer 实例平均分摊消费消息。使用方法：setMessageModel(MessageModel.
Mysql 5.6 Master和Slave 主备切换
04-10 544
--master/slave切换记录，用于后期查找 环境： master: rhel 6.6 mysql 5.6 192.168.15.101 salve: rhel 7.2 mysql 5.6 1...
RocketMQ的autoCreateTopicEnable配置
热门推荐
03-12 2万+
1.使用RocketMQ进行发消息时，必须要指定topic，对于topic的设置有一个开关autoCreateTopicEnable，一般在开发测试环境中会使用默认设置autoCreateTopicEnable = true，但是这样就会导致topic的设置不容易规范管理，没有统一的审核等等，所以在正式环境中会在Broker启动时设置参数autoCreateTopicEnable = false。...
rocketmq 主从同步
08-19
RocketMQ支持主从同步的消息复制模式。在RocketMQ中，一个Broker可以作为Master（主）角色，而另一个Broker则可以作为Slave（从）角色。当消息发送到Master Broker时，Master Broker会将消息复制到所有的Slave ...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值