文章目录
DefaultMQProducer producer = new DefaultMQProducer(“group_name”);
producer.start();通过以上代码完成生产者创建及启动
生产者结构图示
- Producer存在普通生产者和事务生产者
- defaultMQProducerImpl负责消息发送[单向,双向,同步,异步,事务,顺序,普通等]
- 消息轨迹组件持有defaultMQProducerImpl发送消息轨迹
- defaultMQProducerImpl持有hook,发送业务消息产生消息轨迹
源码分析一new DefaultMQProducer
- 设置生产者组
- 构建defaultMQProducerImpl
- 构造函数决定是否启用消息轨迹追踪
public DefaultMQProducer(final String namespace, final String producerGroup, RPCHook rpcHook) {
设置命名空间
this.namespace = namespace;
设置生产者组
this.producerGroup = producerGroup;
构建DefaultMQProducerImpl
defaultMQProducerImpl = new DefaultMQProducerImpl(this, rpcHook);
消息轨迹追踪
if (enableMsgTrace) {
AsyncTraceDispatcher dispatcher = new AsyncTraceDispatcher(producerGroup, TraceDispatcher.Type.PRODUCE, customizedTraceTopic, rpcHook);
dispatcher.setHostProducer(this.defaultMQProducerImpl);
traceDispatcher = dispatcher;
this.defaultMQProducerImpl.registerSendMessageHook(
new SendMessageTraceHookImpl(traceDispatcher));
}
}
- DefaultMQProducerImpl组件负责消息发送
- 发送oneway,双向[同步,异步],事务,顺序,普通消息
public DefaultMQProducerImpl(final DefaultMQProducer defaultMQProducer, RPCHook rpcHook) {
设置 DefaultMQProducer 可以鉴别普通生产者还是事务生产者 生产者组等信息
this.defaultMQProducer = defaultMQProducer;
this.rpcHook = rpcHook;
异步发送的线程池队列
this.asyncSenderThreadPoolQueue = new LinkedBlockingQueue<Runnable>(50000);
异步发送时使用的线程池
this.defaultAsyncSenderExecutor = new ThreadPoolExecutor(
Runtime.getRuntime().availableProcessors(),
Runtime.getRuntime().availableProcessors(),
1000 * 60,
TimeUnit.MILLISECONDS,
this.asyncSenderThreadPoolQueue,
new ThreadFactory() {
private AtomicInteger threadIndex = new AtomicInteger(0);
@Override
public Thread newThread(Runnable r) {
return new Thread(r, "AsyncSenderExecutor_" + this.threadIndex.incrementAndGet());
}
});
}
源码分析一start
- 拼接命名空间和分组名称作为最终分组名称
- 启动defaultMQProducerImpl
- 启动消息轨迹
public void start() throws MQClientException {
/-----------------------------------------------------
rmq_sys_ 为系统topic前缀 CID_RMQ_SYS_为内部消费者组前缀 topic=TBW102三种情况则直接返回系统资源,没有命名空间
%DLQ% 或者%RETRY% 打头 则剔除该前缀后判断有没有命名空间 有则不在处理添加的命名空间 没有则加上传入的命令空间
常见格式
namespace%group
group
%RETRY%namespace%group
%DLQ%namespace%group
/----------------------------------------------------
使用%拼接命名空间和生产者组,该规则亦适用消费者组命名
this.setProducerGroup(withNamespace(this.producerGroup));
启动defaultMQProducerImpl
this.defaultMQProducerImpl.start();
默认为null 消息轨迹
if (null != traceDispatcher) {
try {
traceDispatcher.start(this.getNamesrvAddr(), this.getAccessChannel());
} catch (MQClientException e) {
log.warn("trace dispatcher start failed ", e);
}
}
}
源码分析一defaultMQProducerImpl.start
- step-1: 检查配置: 要求group名称必须存在且合理
- step-2: 获取mqClientInstance
- step-3: 注册当前生产者到客户端实例中 producerTable
- step-4: 为默认的topic创建一个 TopicPublishInfo
- step-5: 启动客户端通信实例[包含诸多task处理: 因消费者与生产者通信层共用MQClientInstance实例,所以包含消费者相关task处理]
- step-6: 轮询broker地址向所有broker发送心跳
public void start(final boolean startFactory) throws MQClientException {
switch (this.serviceState) {
case CREATE_JUST:
this.serviceState = ServiceState.START_FAILED;
step-1: 检查配置: 要求group名称必须存在且合理
this.checkConfig();
设置producer实例名称为进程id
if (!this.defaultMQProducer.getProducerGroup().equals(MixAll.CLIENT_INNER_PRODUCER_GROUP)) {
this.defaultMQProducer.changeInstanceNameToPID();
}
step-2: 获取mq客户端管理器[存在缓存] 封装netty
this.mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(this.defaultMQProducer, rpcHook);
step-3: 注册当前生产者到客户端实例中 producerTable《group producer》
mqclient监听nameserver[如集群变更,能定位到具体生产者组进行topicPublishInfoTable更新]
boolean registerOK = mQClientFactory.registerProducer(this.defaultMQProducer.getProducerGroup(), this);
if (!registerOK) {
this.serviceState = ServiceState.CREATE_JUST;
throw new MQClientException("......"); // 删除异常内容
}
step-4: 为默认的topic创建一个 TopicPublishInfo[包含topic在broker端信息,用于消息发送][通过nameserver服务发现获取TopicPublishInfo内容]
this.topicPublishInfoTable.put(this.defaultMQProducer.getCreateTopicKey(), new TopicPublishInfo());
if (startFactory) {
step-5: 启动客户端通信实例 启动netty 定时任务 启动拉取消息服务和rebalanceService服务
拉取nameserver的routeinfomanager存储信息 从而持有broker相关信息
mQClientFactory.start();
}
...... 删除日志
this.serviceState = ServiceState.RUNNING;
break;
case RUNNING:
case START_FAILED:
case SHUTDOWN_ALREADY:
throw new MQClientException("The producer service state not OK, maybe started once, "
+ this.serviceState
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
null);
default:
break;
}
step-6: 向broker发送心跳
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
...... 删除其他代码
}
start创建通信工具一 getOrCreateMQClientInstance
- 组件间注入处理: producerImpl持有MqClientInstance
- MqClientInstance负责管控与nameserver和broker的消息调用
组件关系图一getOrCreateMQClientInstance
源码分析一getOrCreateMQClientInstance
- 根据ip等元信息从缓存获取MQClientInstance
- 缓存不存在实例化MQClientInstance
public MQClientInstance getOrCreateMQClientInstance(final ClientConfig clientConfig, RPCHook rpcHook) {
与ip强相关
String clientId = clientConfig.buildMQClientId();
缓存存在从缓存获取
MQClientInstance instance = this.factoryTable.get(clientId);
实例化MQClientInstance
if (null == instance) {
instance =
new MQClientInstance(clientConfig.cloneClientConfig(),
this.factoryIndexGenerator.getAndIncrement(), clientId, rpcHook);
MQClientInstance prev = this.factoryTable.putIfAbsent(clientId, instance);
if (prev != null) {
instance = prev;
log.warn("Returned Previous MQClientInstance for clientId:[{}]", clientId);
} else {
log.info("Created new MQClientInstance for clientId:[{}]", clientId);
}
}
return instance;
}
实例化MQClientInstance
- step-1: 创建消息处理器 接收并处理request,response消息
- step-2: 更新下本地的nameserver地址
- step-3: 管理命令的通信
- step-4: 消费者拉取消息线程
- step-5: 消费者负载均衡再平衡线程
- step-6: 内部默认组 当消息处理失败且重发broker时再次失败 则使用defaultMQProducer再次发送broker进行兜底
- step-7: 统计相关
public MQClientInstance(ClientConfig clientConfig, int instanceIndex, String clientId, RPCHook rpcHook) {
this.clientConfig = clientConfig;
this.instanceIndex = instanceIndex;
this.nettyClientConfig = new NettyClientConfig();
this.nettyClientConfig.setClientCallbackExecutorThreads(clientConfig.getClientCallbackExecutorThreads());
this.nettyClientConfig.setUseTLS(clientConfig.isUseTLS());
step-1: 创建消息处理器 接收并处理request,response消息
this.clientRemotingProcessor = new ClientRemotingProcessor(this);
与broker[所有节点],nameserver通信 必须传递addr 明确对方地址
this.mQClientAPIImpl = new MQClientAPIImpl(this.nettyClientConfig, this.clientRemotingProcessor, rpcHook, clientConfig);
step-2: 更新下本地的nameserver地址
if (this.clientConfig.getNamesrvAddr() != null) {
this.mQClientAPIImpl.updateNameServerAddressList(this.clientConfig.getNamesrvAddr());
log.info("user specified name server address: {}", this.clientConfig.getNamesrvAddr());
}
this.clientId = clientId;
step-3: 管理命令的通信
this.mQAdminImpl = new MQAdminImpl(this);
step-4: 消费者拉取消息线程
this.pullMessageService = new PullMessageService(this);
step-5: 消费者负载均衡再平衡线程
this.rebalanceService = new RebalanceService(this);
step-6: 内部默认组 当消息处理失败且重发broker时再次失败 则使用defaultMQProducer再次发送broker进行兜底
this.defaultMQProducer = new DefaultMQProducer(MixAll.CLIENT_INNER_PRODUCER_GROUP);
this.defaultMQProducer.resetClientConfig(clientConfig);
step-7: 统计相关
this.consumerStatsManager = new ConsumerStatsManager(this.scheduledExecutorService);
...... 删除日志
}
start一向MQClientInstance注册分组名和生产者
- MQClientInstance会定时拉取nameserver上的broker注册信息
- 当信息发生变化 会通过producerTable获取生产者并同步注册信息
---------- MQClientInstance.registerProducer----------
public boolean registerProducer(final String group, final DefaultMQProducerImpl producer) {
添加group和producer到producerTable
MQProducerInner prev = this.producerTable.putIfAbsent(group, producer);
return true;
}
start一MQClientFactory启动
- step-1: 组装netty Client模板[注意并没有向任何服务器发起连接]
- step-2: 启动所有的定时任务
- step-3: 消费者:推拉的本质都是拉消息,开启拉取消息线程
- step-4: 消费者负载均衡
- step-5: 如果已经启动 则终止
public void start() throws MQClientException {
synchronized (this) {
switch (this.serviceState) {
case CREATE_JUST:
this.serviceState = ServiceState.START_FAILED;
if (null == this.clientConfig.getNamesrvAddr()) {
获取nameserver 另外client也会定时刷新
this.mQClientAPIImpl.fetchNameServerAddr();
}
step-1: 组装netty Client模板[注意并没有向任何服务器发起连接]
this.mQClientAPIImpl.start();
step-2: 启动所有的定时任务
this.startScheduledTask();
step-3: 消费者:推拉的本质都是拉消息,开启拉取消息线程
this.pullMessageService.start();
step-4: 消费者负载均衡
this.rebalanceService.start();
step-5: 如果已经启动 则终止
this.defaultMQProducer.getDefaultMQProducerImpl().start(false);
log.info("the client factory [{}] start OK", this.clientId);
this.serviceState = ServiceState.RUNNING;
break;
case START_FAILED:
throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
default:
break;
}
}
}
总结
- 一个进程含有生产者和消费者n个,但只含有一个mqClient通信socketchannel,并且生产者和消费者共享
扩展点一通信层定时任务
1 应用程序进程在通信层上架设生产者组和消费者组集合[参加下图]
2 broker端注册信息到nameserver
3 nameserver内存储的broker信息可能会发生变更
4 MqClientInstance负责定时拉取nameserver上的注册信息,如果发生变化则修改上层producer 和consumer的相关元信息
- step-1: 获取nameserver 每隔2分钟尝试获取一次NameServer地址
- step-2: 每隔30S尝试更新主题路由信息
- step-3: 每隔30S 进行Broker心跳检测
- step-4: 默认每隔5秒持久化ConsumeOffset
private void startScheduledTask() {
step-1: 获取nameserver 每隔2分钟尝试获取一次NameServer地址
if (null == this.clientConfig.getNamesrvAddr()) {
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
MQClientInstance.this.mQClientAPIImpl.fetchNameServerAddr();
} catch (Exception e) {
log.error("ScheduledTask fetchNameServerAddr exception", e);
}
}
}, 1000 * 10, 1000 * 60 * 2, TimeUnit.MILLISECONDS);
}
step-2: 每隔30S尝试更新主题路由信息
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
// 更新路由信息
try {
MQClientInstance.this.updateTopicRouteInfoFromNameServer();
} catch (Exception e) {
log.error("ScheduledTask updateTopicRouteInfoFromNameServer exception", e);
}
}
}, 10, this.clientConfig.getPollNameServerInterval(), TimeUnit.MILLISECONDS);
step-3: 每隔30S 进行Broker心跳检测
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
清除挂掉的broker
try {
MQClientInstance.this.cleanOfflineBroker();
MQClientInstance.this.sendHeartbeatToAllBrokerWithLock();
} catch (Exception e) {
log.error("ScheduledTask sendHeartbeatToAllBroker exception", e);
}
}
}, 1000, this.clientConfig.getHeartbeatBrokerInterval(), TimeUnit.MILLISECONDS);
step-4: 默认每隔5秒持久化ConsumeOffset
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
/**
* 消费者 持久化消费进度
* 广播 消费端
* 集群 broker端
*/
try {
MQClientInstance.this.persistAllConsumerOffset();
} catch (Exception e) {
log.error("ScheduledTask persistAllConsumerOffset exception", e);
}
}
}, 1000 * 10, this.clientConfig.getPersistConsumerOffsetInterval(), TimeUnit.MILLISECONDS);
...... 删除自动调整线程池代码
}
扩展点一NettyRemotingClient.start
- 代码不做注释,可以看出仅仅构建netty模板,但无实际connect
- 因为MQClientInstance需要和所有的broker节点通信,需要上层指定地址(地址从nameserver获取,nameserver地址通过静态配置或者静态服务器配置获取)
- 只有在获取channel的时候,才会调用bootstrap.connect
public void start() {
...... 删除其他代码
Bootstrap handler = this.bootstrap.group(this.eventLoopGroupWorker).channel(NioSocketChannel.class)
.option(ChannelOption.TCP_NODELAY, true)
.option(ChannelOption.SO_KEEPALIVE, false)
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, nettyClientConfig.getConnectTimeoutMillis())
.option(ChannelOption.SO_SNDBUF, nettyClientConfig.getClientSocketSndBufSize())
.option(ChannelOption.SO_RCVBUF, nettyClientConfig.getClientSocketRcvBufSize())
.handler(new ChannelInitializer<SocketChannel>() {
@Override
public void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
if (nettyClientConfig.isUseTLS()) {
if (null != sslContext) {
pipeline.addFirst(defaultEventExecutorGroup, "sslHandler", sslContext.newHandler(ch.alloc()));
log.info("Prepend SSL handler");
} else {
log.warn("Connections are insecure as SSLContext is null!");
}
}
pipeline.addLast(
defaultEventExecutorGroup,
new NettyEncoder(),
new NettyDecoder(),
new IdleStateHandler(0, 0, nettyClientConfig.getClientChannelMaxIdleTimeSeconds()),
new NettyConnectManageHandler(),
new NettyClientHandler());
}
});
...... 删除其他代码
}
扩展点一关于再平衡以及消息拉取
主要涉及rebalanceService,pullMessageService,defaultMQProducer
于消息消费章节详解
名称 | 作用 |
---|---|
rebalanceService再平衡线程 | 消费者组有多个实例,决定每个消费者实例消费主题的哪些MessageQueue 消息 |
pullMessageService消息拉取线程 | 根据在平衡结果拉取消息,用来进行消费 |
defaultMQProducer兜底发送者 | 消息消费失败,重发broker,mqClientApiImpl重发失败则采用兜底发送者发送 |