问题分析
ClientID是每个连接Rocketmq的唯一标识
如果一个消费者组的instanceName相同,就会造成消费队列分布不均,消费重复,消息堆积等问题。
当客户端clientId生成一致时,出现问题也就很正常了。
那么为什么docker容器使用host模式,会出现问题?
1:dokcer内consumer的pid发现都是1
(由于每台容器都是干净的,只跑一个程序,所以pid都是1)
2:ClientID生成规则可能会用到pid
3:docker容器host模式是共用宿主机网络
所以当pid完全相同,docker容器ip也相同时,就会造成ClientID一样。
原始clientid的instanceName
初始化的时候首先会获取系统属性 rocketmq.client.name 是否有值
如果没有就是用默认值DEFAULT
然后 consumer 启动的时候会判断这参数值是否为 DEFAULT ,如果是的话就调用 UtilAll.getPid()
org.apache.rocketmq.client.ClientConfig#changeInstanceNameToPID
Spring封装clientid的instanceName
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-spring-boot-starter</artifactId>
</dependency>
RocketMQUtil.getInstanceName(nameServer)
instanceName生成
clientId组成
clientId的最终组成,org.apache.rocketmq.client.ClientConfig#buildMQClientId
原生消费者生成和Spring封装的生成,区别就在instanceName上面
问题解决
本人使用的是Spring封装的,所以在DefaultRocketMQListenerContainer生成创建完成之后,采取了spring拓展点,进行二次改造。使用时间在原先的基础上拼接组成新的instanceName
@Component
public class RocketMqConsumerPostProcessor implements BeanPostProcessor {
/**
* Apply this {@code BeanPostProcessor} to the given new bean instance <i>after</i> any bean
* initialization callbacks (like InitializingBean's {@code afterPropertiesSet}
* or a custom init-method). The bean will already be populated with property values.
* The returned bean instance may be a wrapper around the original.
* <p>In case of a FactoryBean, this callback will be invoked for both the FactoryBean
* instance and the objects created by the FactoryBean (as of Spring 2.0). The
* post-processor can decide whether to apply to either the FactoryBean or created
* objects or both through corresponding {@code bean instanceof FactoryBean} checks.
* <p>This callback will also be invoked after a short-circuiting triggered by a
* {@link InstantiationAwareBeanPostProcessor#postProcessBeforeInstantiation} method,
* in contrast to all other {@code BeanPostProcessor} callbacks.
* <p>The default implementation returns the given {@code bean} as-is.
*
* @param bean the new bean instance
* @param beanName the name of the bean
* @return the bean instance to use, either the original or a wrapped one;
* if {@code null}, no subsequent BeanPostProcessors will be invoked
* @throws BeansException in case of errors
* @see InitializingBean#afterPropertiesSet
* @see FactoryBean
*/
@Override
public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException {
// 在bean初始化完成后执行
if (bean instanceof DefaultRocketMQListenerContainer) {
DefaultRocketMQListenerContainer defaultRocketMqListenerContainer = (DefaultRocketMQListenerContainer) bean;
DefaultMQPushConsumer consumer = defaultRocketMqListenerContainer.getConsumer();
String instanceName = consumer.getInstanceName();
consumer.setInstanceName(instanceName + "#" + System.nanoTime());
}
return bean;
}
}
在bean初始化完成之后,使用Spring的拓展点功能,对bean进行二次修改,修改consumer的instanceName,确保其唯一性。这样生成的clientId在Rocketmq是唯一的,不会重复。
如果是原始调用修改就更简单,直接获取DefaultMQPushConsumer修改instanceName
改完后的效果如下:
spring封装源码跟踪
RocketMQAutoConfiguration
org.apache.rocketmq.spring.support.DefaultRocketMQListenerContainer