RocketMq是阿里巴巴2016年开源的,贡献给了apache,用java语言开发
优点:
1、解耦
2、削峰
3、数据分发
缺点:
1、系统可用性降低
如果MQ宕机了,就会造成影响
2、系统复杂性提高
引入消息同步问题,顺序问题,消息丢失问题
3、一致性问题
如果消息订阅方一个系统处理失败了,几个订阅方的数据就会造成不一致
市场上各种MQ比较
ActiveMQ RabbitMQ RocketMq kafka
安装
去官网下载RocketMq
解压后启动
> nohup sh bin/mqnamesrv &
> tail -f ~/logs/rocketmqlogs/namesrv.log
The Name Server boot success...
> nohup sh bin/mqbroker -n localhost:9876 &
> tail -f ~/logs/rocketmqlogs/broker.log
The broker[%s, 172.30.30.233:10911] boot success...
注:启动时如果出现启动不成功
去bin/nohup.out查看错误原因
ERROR: Please set the JAVA_HOME variable in your environment, We need java(x64)! !!
停止
> sh bin/mqshutdown broker
The mqbroker(36695) is running...
Send shutdown request to mqbroker(36695) OK
> sh bin/mqshutdown namesrv
The mqnamesrv(36664) is running...
Send shutdown request to mqnamesrv(36664) OK
角色
nameServer是broker的管理者
broker暂存和传输消息,主动上报nameServer
producer生产者发送消息,询问nameServer,指定broker
consumer消费者消费消息,询问nameServer,指定broker
topic区分消息的种类
messageQueue相当于Topic的分分区,用于并行发送和接收消息
集群搭建
nameServer集群是无状态的,每个节点内容都是一样的。
broker会给每一个nameServer节点上报信息
broker集群中主节点负责写,读负责读
broker的master和slave节点通过brokerName确定是否为同一组, id为0是主节点,其他为从节点
producer会和NameServer集群中一个随机节点建立长连接,定期从nameServer取topic路由信息,并向提供topic服务的Mater建立长连接,定时向master发送心跳。producer是完全无状态的,可集群部署
consumer会和NameServer集群中一个随机节点建立长连接,定期从nameServer取topic路由信息,并向提供topic服务的Mater、slave建立长连接,定时向master、slave发送心跳。consumer既可以从master订阅消息,也可以从slave订阅消息,订阅规则由broker决定
主要介绍broker集群模式
1、单master模式
2、多master模式
3、多master和多slave(异步)
4、多master和多slave(同步)
集群工作流程
1、启动NameServer,等待broker、producer、consumer连上来。
2、启动broker,与nameServer建立长连接。发送心跳包,心跳包中包含当前broker信息(broker的IP+端口以及存储的所有topic信息),注册成功后,NameServer中就要broker和Topic的映射关系了
3、手法消息前,创建topic,指定该topic要存储在哪些broker上,也可以在发送消息时自动创建
4、producer发送消息,启动时先跟nameServer集群中的一个节点建立长连接,获取当前发送的topic存在哪些broker上,轮询队列,选择一个队列,然后与该队列所在的broker建立长连接,从而向broker发送消息
5、consumer跟producer相似,从NameServer中一个节点建立长连接,然后确定当前订阅的topic所在的broker,然后直接建立连接,开始消费
集群搭建
注意点,broker配置文件设置,根据选择的集群模式,选择对应的配置文件,然后用对应的配置文件启动集群
rocketMq-console集群监控。
在git上下载代码,修改NameServer配置后,打包后放到集群服务上。然后java -jar ***.jar后启动
消息发送者步骤
1、创建生产者producer,定制组名
2、指定NameServer地址
3、启动producer
4、创建消息对象,指定主题topic、tag和消息体
5、发送消息
6、关闭生产者
消息消费者步骤
1、创建消费者consumer,指定组名
2、指定NameServer地址
3、订阅主题topic和tag
4、设置回调函数,处理消息
5、启动消费者consumer
案例
添加依赖
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-client</artifactId>
<version>4.3.0</version>
</dependency>
同步发送
生产者
public class SyncProducer {
public static void main(String[] args) throws Exception {
//Instantiate with a producer group name.
DefaultMQProducer producer = new
DefaultMQProducer("please_rename_unique_group_name");
// Specify name server addresses.
producer.setNamesrvAddr("localhost:9876");
//Launch the instance.
producer.start();
for (int i = 0; i < 100; i++) {
//Create a message instance, specifying topic, tag and message body.
Message msg = new Message("TopicTest" /* Topic */,
"TagA" /* Tag */,
("Hello RocketMQ " +
i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body */
);
//Call send message to deliver message to one of brokers.
SendResult sendResult = producer.send(msg);
System.out.printf("%s%n", sendResult);
}
//Shut down once the producer instance is not longer in use.
producer.shutdown();
}
}
异步发送
生产者
public class AsyncProducer {
public static void main(String[] args) throws Exception {
//Instantiate with a producer group name.
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
// Specify name server addresses.
producer.setNamesrvAddr("localhost:9876");
//Launch the instance.
producer.start();
producer.setRetryTimesWhenSendAsyncFailed(0);
for (int i = 0; i < 100; i++) {
final int index = i;
//Create a message instance, specifying topic, tag and message body.
Message msg = new Message("TopicTest",
"TagA",
"OrderID188",
"Hello world".getBytes(RemotingHelper.DEFAULT_CHARSET));
producer.send(msg, new SendCallback() {
@Override
public void onSuccess(SendResult sendResult) {
System.out.printf("%-10d OK %s %n", index,
sendResult.getMsgId());
}
@Override
public void onException(Throwable e) {
System.out.printf("%-10d Exception %s %n", index, e);
e.printStackTrace();
}
});
}
//Shut down once the producer instance is not longer in use.
producer.shutdown();
}
}
单向发送
生产者
public class OnewayProducer {
public static void main(String[] args) throws Exception{
//Instantiate with a producer group name.
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
// Specify name server addresses.
producer.setNamesrvAddr("localhost:9876");
//Launch the instance.
producer.start();
for (int i = 0; i < 100; i++) {
//Create a message instance, specifying topic, tag and message body.
Message msg = new Message("TopicTest" /* Topic */,
"TagA" /* Tag */,
("Hello RocketMQ " +
i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body */
);
//Call send message to deliver message to one of brokers.
producer.sendOneway(msg);
}
//Shut down once the producer instance is not longer in use.
producer.shutdown();
}
}
消费者
public class Consumer {
public static void main(String[] args) throws InterruptedException, MQClientException {
// Instantiate with specified consumer group name.
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name");
// Specify name server addresses.
consumer.setNamesrvAddr("localhost:9876");
// Subscribe one more more topics to consume.
consumer.subscribe("TopicTest", "*");
// Register callback to execute on arrival of messages fetched from brokers.
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
//Launch the consumer instance.
consumer.start();
System.out.printf("Consumer Started.%n");
}
}
消费者广播模式和负载均衡模式
设置模式
consumer.setMessageModel(MessageModel.CLUSTERING);//默认为负载均衡模式
consumer.setMessageModel(MessageModel.BROADCASTING);//广播模式
保证消息顺序
例如:订单的顺序流程:创建、付款、推送、完成,订单号相同的消息会被先后发送到同一个队列中,消费时,同一个orderId获取到的肯定是同一个队列.
生产者添加如下发送代码
//根据orderId取模顺序将相同orderId消息发放送到同一个队列
int orderId = 10001;//模拟一个订单ID
producer.send(msg, new MessageQueueSelector() {
@Override
public MessageQueue select(List<MessageQueue> list, Message message, Object o) {
int id = (int)o;
int i = id%list.size();
return list.get(i);
}
},orderId);
消费者添加如下代码
consumer.registerMessageListener(new MessageListenerOrderly() {
@Override
public ConsumeOrderlyStatus consumeMessage(List<MessageExt> list, ConsumeOrderlyContext consumeOrderlyContext) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), list);
return ConsumeOrderlyStatus.SUCCESS;
}
});
预定消息(也叫延迟消息)
//组装消息体
Message msg = new Message("TopicTest" /* Topic */,
"TagA" /* Tag */,
("Hello RocketMQ " +
i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body */
);
//设置延时等级
msg.setDelayTimeLevel(1);
批量发送消息
List<Message> messageList = new ArrayList();
//构建多条消息
Message msg1 = new Message();
Message msg2 = new Message();
Message msg3 = new Message();
messageList.add(msg1);
messageList.add(msg2);
messageList.add(msg3);
当批量发送消息过大,如邮件大小不超过超过1M则用如下代码分割
public class ListSplitter implements Iterator<List<Message>> {
private final int SIZE_LIMIT = 1000 * 1000;
private final List<Message> messages;
private int currIndex;
public ListSplitter(List<Message> messages) {
this.messages = messages;
}
@Override public boolean hasNext() {
return currIndex < messages.size();
}
@Override public List<Message> next() {
int nextIndex = currIndex;
int totalSize = 0;
for (; nextIndex < messages.size(); nextIndex++) {
Message message = messages.get(nextIndex);
int tmpSize = message.getTopic().length() + message.getBody().length;
Map<String, String> properties = message.getProperties();
for (Map.Entry<String, String> entry : properties.entrySet()) {
tmpSize += entry.getKey().length() + entry.getValue().length();
}
tmpSize = tmpSize + 20; //for log overhead
if (tmpSize > SIZE_LIMIT) {
//it is unexpected that single message exceeds the SIZE_LIMIT
//here just let it go, otherwise it will block the splitting process
if (nextIndex - currIndex == 0) {
//if the next sublist has no element, add this one and then break, otherwise just break
nextIndex++;
}
break;
}
if (tmpSize + totalSize > SIZE_LIMIT) {
break;
} else {
totalSize += tmpSize;
}
}
List<Message> subList = messages.subList(currIndex, nextIndex);
currIndex = nextIndex;
return subList;
}
}
//then you could split the large list into small ones:
ListSplitter splitter = new ListSplitter(messages);
while (splitter.hasNext()) {
try {
List<Message> listItem = splitter.next();
producer.send(listItem);
} catch (Exception e) {
e.printStackTrace();
//handle the error
}
}
消息过滤
tag过滤
消费者在消费的时候,指定tag
//同时消费tag1和tag2
consumer.subscribe("topicTest","tag1 || tag2");
//消费所有topic为“topicTest”消息
consumer.subscribe("topicTest","*");
sql语法过滤
//生产者自定义消息属性
msg.putUserProperty("a",String.valueOf(i));
//消费者根据条件过滤消息去消费
consumer.subscribe("TopicTest", MessageSelector.bySql("a>5"));//根据自定义条件过滤消息
事务消息
应用
1,交易状态
事务消息有三种状态:
(1)TransactionStatus.CommitTransaction:提交事务,表示允许使用者使用此消息。
(2)TransactionStatus.RollbackTransaction:回滚事务,表示该消息将被删除且不允许使用。
(3)TransactionStatus.Unknown:中间状态,表示需要MQ进行回溯以确定状态。
2,发送交易信息
创建事务生产方
使用TransactionMQProducer类创建生产方客户端,并指定唯一的producerGroup。执行本地事务后,需要根据执行结果对MQ进行回复,回复状态如上节所述。
package com.qing.zhao.common;
import org.apache.commons.lang3.StringUtils;
import org.apache.rocketmq.client.producer.LocalTransactionState;
import org.apache.rocketmq.client.producer.TransactionListener;
import org.apache.rocketmq.client.producer.TransactionMQProducer;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.remoting.common.RemotingHelper;
public class TransactionMqProducer {
public static void main(String[] args) throws Exception {
TransactionMQProducer mqProducer = new TransactionMQProducer("group");
mqProducer.setNamesrvAddr("localhost:9876");
mqProducer.setTransactionListener(new TransactionListener() {
@Override
public LocalTransactionState executeLocalTransaction(Message message, Object o) {
if (StringUtils.equals("tag1",message.getTags())){
return LocalTransactionState.COMMIT_MESSAGE;
}else if (StringUtils.equals("tag2",message.getTags())){
return LocalTransactionState.ROLLBACK_MESSAGE;
}else if (StringUtils.equals("tag3",message.getTags())){
return LocalTransactionState.UNKNOW;
}
return LocalTransactionState.UNKNOW;
}
@Override
public LocalTransactionState checkLocalTransaction(MessageExt messageExt) {
System.out.println("消息的tag为:"+messageExt.getTags());
return LocalTransactionState.COMMIT_MESSAGE;
}
});
String[] tag = new String[]{"tag1","tag2","tag3"};
mqProducer.start();
for (int i=0; i<3; i++){
Message msg = new Message("TopicTest" /* Topic */,
tag[i] /* Tag */,
("Hello RocketMQ "+i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body */
);
mqProducer.sendMessageInTransaction(msg,null);
}
}
}