RabbitMQ中的死信交换器(dead letter exchange)可以接收下面三种场景中的消息:
- 消费者对消息使用了
basicReject或者basicNack回复,并且requeue参数设置为false,即不再将该消息重新在消费者间进行投递. - 消息在队列中超时. RabbitMQ可以在单个消息或者队列中设置
TTL属性. - 队列中的消息已经超过其设置的最大消息个数.
死信交换器可以在程序中设置,也可以使用rabbitmqctl工具进行设置.关于死信交换器的介绍请参考RabbitMQ官网https://www.rabbitmq.com/dlx.html.
死信交换器可以统计或者监控错误消息, 下面使用RabbitMQ的Java客户端库给出一个完整的例子, 结合第一种消息来源说明死信交换器的使用.
示例代码
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;
import com.rabbitmq.client.AMQP.BasicProperties;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.DeliverCallback;
import com.rabbitmq.client.Envelope;
public class DeadLetterExchange {
private static final String NORMAL_EXCHANGE = "normal_exchange";
private static final String DEAD_LETTER_EXCHANGE = "dead_letter_exchange";
private static final String DEFAULT_WORKER_QUEUE = "queue_to_normal_exchange";
private static final String DLX_WORKER_QUEUE = "queue_to_deadletter_exchange";
private static final String NORMAL_ROUTE_KEY = "normal_route_key";
private static final String DLX_ROUTE_KEY = "dlx_route_key";
private Connection connection;
// For consumer and producer schedule.
private final ExecutorService executorService;
private static class SimpleThreadFactory implements ThreadFactory {
private final AtomicInteger idx = new AtomicInteger(0);
public Thread newThread(Runnable r) {
return new Thread(r, "Thread#" + idx.getAndIncrement());
}
}
private static class WorkerConsumer implements Runnable {
private final Connection connection;
private Channel channel;
private String queue;
private final DeliverCallback deliverCallback = (consumerTag, delivery) -> {
final String message = new String(delivery.getBody(), "UTF-8");
final Envelope envelope = delivery.getEnvelope();
final long deliveryTag = envelope.getDeliveryTag();
final boolean redelivery = envelope.isRedeliver();
final BasicProperties props = delivery.getProperties();
final String msgId = props.getMessageId();
// Reject some message and not requeue it.
final boolean reject = (deliveryTag % 3) == 2;
System.out.println("Received normal exchange msg:" + message
+ " msg id:" + msgId
+ " reject:" + reject);
if (reject ) {
channel.basicReject(deliveryTag, false);
} else {
channel.basicAck(deliveryTag, false);
}
};
public WorkerConsumer(Connection connection, String queue) {
this.connection = connection;
this.queue = queue;
}
@Override
public void run() {
try {
channel = connection.createChannel();
channel.exchangeDeclare(NORMAL_EXCHANGE, "direct");
final Map<String, Object> args = new HashMap<>();
args.put("x-dead-letter-exchange", DEAD_LETTER_EXCHANGE);
if (queue == null) {
queue = DEFAULT_WORKER_QUEUE;
}
channel.queueDeclare(queue, false, false, true, args);
channel.queueBind(queue, NORMAL_EXCHANGE, NORMAL_ROUTE_KEY);
final String tag = channel.basicConsume(queue, false, deliverCallback, consumerTag -> {System.out.println("Cancel worker consumer:" +consumerTag);});
System.out.println("worker consumer tag:" + tag);
} catch (IOException e) {
e.printStackTrace();
}
}
}
// This special consumer handle messages delivered to dead letter exchange.
private static class DLXConsumer implements Runnable {
private final Connection connection;
private final DeliverCallback deliverCallback = (consumerTag, delivery) -> {
final String message = new String(delivery.getBody(), "UTF-8");
final BasicProperties props = delivery.getProperties();
final String msgId = props.getMessageId();
System.out.println("Received dead letter exchange msg:" + message
+ " msg id:" + msgId);
};
public DLXConsumer(Connection connection) {
this.connection = connection;
}
@Override
public void run() {
try {
final Channel channel = connection.createChannel();
channel.exchangeDeclare(DEAD_LETTER_EXCHANGE, "direct");
channel.queueDeclare(DLX_WORKER_QUEUE, false, false, true, null);
channel.queueBind(DLX_WORKER_QUEUE, DEAD_LETTER_EXCHANGE, NORMAL_ROUTE_KEY);
final String tag = channel.basicConsume(DLX_WORKER_QUEUE, true, deliverCallback, consumerTag -> {System.out.println("Cancel DLX consumer:" +consumerTag);});
System.out.println("dlx consumer tag:" + tag);
} catch (IOException e) {
e.printStackTrace();
}
}
}
private static class Producer implements Runnable {
private static final int MSG_COUNTS = 20;
private final Connection connection;
public Producer(Connection connection) {
this.connection = connection;
}
@Override
public void run() {
try {
final Channel channel = connection.createChannel();
// Send some messages, some of them can not route to any queue.
for (int i = 0; i < MSG_COUNTS; i++) {
final BasicProperties.Builder propsBuilder = new BasicProperties.Builder();
final BasicProperties props = propsBuilder.appId(Thread.currentThread().toString()).messageId(String.valueOf(i)).build();
final boolean useInvalidRouteKey = (i % 4 == 3);
if (!useInvalidRouteKey) {
channel.basicPublish(NORMAL_EXCHANGE, NORMAL_ROUTE_KEY, props, "[normal message.]".getBytes());
} else {
channel.basicPublish(NORMAL_EXCHANGE, "not_existed_route_key", props, "[lost message.]".getBytes());
}
}
} catch(IOException e) {
e.printStackTrace();
}
}
}
public DeadLetterExchange() {
executorService = Executors.newCachedThreadPool(new SimpleThreadFactory());
final ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("localhost");
try {
connection = connectionFactory.newConnection();
} catch (IOException | TimeoutException e) {
e.printStackTrace();
}
}
public void execWorkderConsumer(String queue) {
executorService.submit(new WorkerConsumer(connection, queue));
}
public void execDLXConsumer() {
executorService.submit(new DLXConsumer(connection));
}
public void execProducer(int n) {
for (int i = 0; i < n; i++) {
executorService.submit(new Producer(connection));
}
}
public void clearConfig(String[] queues) {
try {
final Channel channel = connection.createChannel();
channel.exchangeDelete(DEAD_LETTER_EXCHANGE);
channel.exchangeDelete(NORMAL_EXCHANGE);
channel.queueDelete(DEFAULT_WORKER_QUEUE);
channel.queueDelete(DLX_WORKER_QUEUE);
for (String queue : queues) {
channel.queueDelete(queue);
}
System.out.println("Config cleared.");
System.exit(0);
} catch(IOException e) {
e.printStackTrace();
}
}
public static void main(String args[]) {
if (args.length > 0) {
final String which = args[0];
if ("consumer".equals(which)) {
final String workQueue = (args.length > 1 ? args[1] : null);
new DeadLetterExchange().execWorkderConsumer(workQueue);
} else if ("producer".equals(which)) {
final int n = (args.length > 1 ? Integer.parseInt(args[1]) : 1);
new DeadLetterExchange().execProducer(n);
} else if ("dlxconsumer".equals(which)) {
new DeadLetterExchange().execDLXConsumer();
} else if ("clearconfig".equals(which)) {
String[] queues = new String[0];
if (args.length > 1) {
queues = new String[args.length - 1];
System.arraycopy(args, 1, queues, 0, args.length - 1);
}
new DeadLetterExchange().clearConfig(queues);
}
}
}
}
其中,producer投递消息的交换器定义为NORMAL_EXCHANGE, 其consumer实现定义为WorkerConsumer, 死信交换器定义为DEAD_LETTER_EXCHANGE, 其consumer实现定义为DLXConsumer.
示例代码中,可以为NORMAL_EXCHANGE绑定实现一个消息队列, 可以运行多个consumer在该队列中以Round-robin的方式处理消息, 也可以为NORMAL_EXCHANGE绑定实现多个队列.通常死信交换器上的消息处理负载可能并不高,可以用一个consumer来处理其消息, 当然为NORMAL_EXCHANGE绑定多个队列或者运行多个consumer也是可以的. 使用死信交换器时, 其架构如下(使用一个worker consumer队列的情况):
首先, 将该类导出为Runnable JAR包.打开一个控制台终端,运行一个worker consumer:
java -jar deadletterexchange.jar consumer
打开另一个控制台终端, 运行一个DLX consumer:
java -jar deadletterexchange.jar dlxconsumer
打开另一个控制台终端, 运行一个producer:
java -jar deadletterexchange.jar producer
producer发送完消息后, worker consumer的log输出为:
Received normal exchange msg:[normal message.] msg id:0 reject:false
Received normal exchange msg:[normal message.] msg id:1 reject:true
Received normal exchange msg:[normal message.] msg id:2 reject:false
Received normal exchange msg:[normal message.] msg id:4 reject:false
Received normal exchange msg:[normal message.] msg id:5 reject:true
Received normal exchange msg:[normal message.] msg id:6 reject:false
Received normal exchange msg:[normal message.] msg id:8 reject:false
Received normal exchange msg:[normal message.] msg id:9 reject:true
Received normal exchange msg:[normal message.] msg id:10 reject:false
Received normal exchange msg:[normal message.] msg id:12 reject:false
Received normal exchange msg:[normal message.] msg id:13 reject:true
Received normal exchange msg:[normal message.] msg id:14 reject:false
Received normal exchange msg:[normal message.] msg id:16 reject:false
Received normal exchange msg:[normal message.] msg id:17 reject:true
Received normal exchange msg:[normal message.] msg id:18 reject:false
DLX consumer的log输出为:
Received dead letter exchange msg:[normal message.] msg id:1
Received dead letter exchange msg:[normal message.] msg id:5
Received dead letter exchange msg:[normal message.] msg id:9
Received dead letter exchange msg:[normal message.] msg id:13
Received dead letter exchange msg:[normal message.] msg id:17
在worker consumer中, 对部分消息使用了basicReject进行拒绝回复, 并且其requeue属性为false. 从worker consumer的log输出看,这些被拒绝的消息ID为:1,5,9,13,17. 再来看DLX consumer的log输出,发现worker consumer上这些被拒绝回复的消息全部路由到了DLX交换器上了.
对于死信交换器需要注意的几个细节是:
- 死信交换器相较于非死信交换器没有任何特殊的属性, 可以为其指定为任何交换器类型(例子中使用了
direct类型).唯一要做的是在定义worker consumer的队列时, 使用参数x-dead-letter-exchange指定死信交换器即可. - 在
worker consumer中定义队列指定死信交换器时, 该死信交换器甚至可以不存在(即还没有定义). 示例中先在终端中运行了worker consumer,然后运行了DLX consumer. 在运行worker consumer时, 死信交换器还没有被定义.但是当有消息需要被路由到死信交换器时, 其必须已经存在. worker consumer可以使用basicReject或者basicNack对消息进行确认, 当这两个命令的requeue参数设置为false时, 该消息才被路由到死信交换器.否则, 消息将被重新在worker consumer间进行投递. 需要注意的是, 重新投递的消息可能还会被投递到拒绝该消息的worker consumer上, 形成循环.- 如果
producer发送消息时, 如果该消息无法路由到任何队列, 那么这个消息是不会被路由到死信交换器的. 示例中producer发送的部分消息,其route key为not_existed_route_key, 这些消息无法路由到任何队列, 也不会被路由到死信交换器. - 默认情况下, 路由到死信交换器的消息,将使用
producer发送时的route key. 但是, 可以改变路由到死信交换器的route key. 在worker consumer定义其队列时, 除了使用x-dead-letter-exchange参数指定死信交换器, 还可以使用x-dead-letter-routing-key参数指定该消息的route key.使用这个配置可以在DLX worker中判断出消息是从哪个worker consumer路由过来的.
BTW: 因为这个示例导出的Runnable JAR包中包含了RabbitMQ的Java客户端库,所以可以直接运行. 否则,运行时需要在classpath中指定其客户端库路径.
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
