Rabbitmq 概述及代码示例

1、概述

amqp（advanced message queuing protocol）协议是一个高级抽象层消息协议，即高级消息队列协议，是应用层协议的一个开放标准，为面向消息的中间价设计。消息中间价主要用于组件之间的解耦，消息的发送者无需知道消息使用者的存在。

amqp的特征是面向消息、队列、路由（包括点对点和发布/订阅）、可靠性和安全性。

2、使用场景

        a、两个系统间需要通过定时任务来同步数据

        b、异构系统的不同进程间相互调用、通讯

        c、生产者消费者模型的使用

3、基本概念

ConnectionFactory、Connection、Channel都是Rabbitmq对外提供的API中最基本的对象。Connection是rabbitmq的socket连接，它封装了socket协议的相关部分逻辑。connectionfactory为Connection的制造工厂。

channel是我们与Rabbitmq打交道的一个借口，我们大部分的业务操作都是在channel这个借口中完成，包括定义queue、发布exchange、绑定queue和exchange、发布消息等。

4、基本消息类型介绍【Queue】

queue是rabbitmq的内部对象，用于存储消息。

rabbitmq中的所有消息都存储在queue中，上图为生产者消费者模型，图一为一个生产者对应一个消费者，图二为一个生产者对应多个（两个）消费者，那么在对应的使用场景下，我们可以应用于这样的场景：A系统和B系统间需要定时通信，如A系统中的一些订单数据需要到B系统中进行实时统计分析，那么将订单数据放入消息队列，B系统从消息队列中读取便实现系统间通讯。场景二：A系统中需要给用户发送打折消息，用户量很大，需要发送时间很长，将需要发送的人员写入消息队列，同时开启多个线程从消息队列中读取人员信息，并发送消息，就是一个生产者对应多个消费者，实现异步处理。

   
   
#!/usr/bin/env python
#生产者端
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(
        host=\'localhost\'))
channel = connection.channel()
channel.queue_declare(queue=\'task_queue\', durable=True)
message = \' \'.join(sys.argv[1:]) or "Hello World!"
channel.basic_publish(exchange=\'\',
                      routing_key=\'task_queue\',
                      body=message,
                      properties=pika.BasicProperties(
                         delivery_mode = 2, # make message persistent
                      ))
print(" [x] Sent %r" % message)
connection.close()

    
    
#!/usr/bin/env python
#消费者端
import pika
import time
connection = pika.BlockingConnection(pika.ConnectionParameters(
        host=\'localhost\'))
channel = connection.channel()
channel.queue_declare(queue=\'task_queue\', durable=True)
print(\' [*] Waiting for messages. To exit press CTRL+C\')
def callback(ch, method, properties, body):
    print(" [x] Received %r" % body)
    time.sleep(body.count(b\'.\'))
    print(" [x] Done")
    ch.basic_ack(delivery_tag = method.delivery_tag)
channel.basic_qos(prefetch_count=1)
channel.basic_consume(callback,
                      queue=\'task_queue\')
channel.start_consuming()

5、Message acknowledge

    在应用场景中，我们可能需要掌握消息是否被消费者正确消费，这时消费者消费完后发送一条消息给生产者，队列知道消息被消费了，从队列中移除。

    但是产生另外一个问题，如果开发人员处理完逻辑后，忘了发送回执为rabbitmq，会导致queue中国堆积消息越来越多，消费者重启后消息重复发送给消费者，导致重复执行，因此要注意。

6、message durability

    如果我们希望rabbitmq重启后，消息不丢失，可以设置queue为持久化的，重启后消息依然存在。

7、prefetch count

    有多个消费者时，会出现问题，有的消费者干的活很多，而有的很闲，因此通过设置prefetchCount=1，使得消息公平分发。

8、exchange

    

消息我们需要通过一个交换器，由路由器将消息分发到不同的消费者上，此时消息可能被分发到一个队列或者多个队列。

9、routing key

    生产者将消息发送给exchang的时候，一般会指定一个routing key，来指定这个消息的路由规则，而这个routing key 需要与exhcange type 及binding key 联合使用才能生效。

生产者将消息发送给exchange时，通过指定routing key来确定消息流流向。

emit_producer.py

      
      
#!/usr/bin/env python
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(
        host=\'localhost\'))
channel = connection.channel()
channel.exchange_declare(exchange=\'logs\',
                         type=\'fanout\')
message = \' \'.join(sys.argv[1:]) or "info: Hello World!"
channel.basic_publish(exchange=\'logs\',
                      routing_key=\'\',
                      body=message)
print(" [x] Sent %r" % message)
connection.close()

receive_log.py

       
       
#!/usr/bin/env python
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(
        host=\'localhost\'))
channel = connection.channel()
channel.exchange_declare(exchange=\'logs\',
                         type=\'fanout\')
result = channel.queue_declare(exclusive=True)
queue_name = result.method.queue
channel.queue_bind(exchange=\'logs\',
                   queue=queue_name)
print(\' [*] Waiting for logs. To exit press CTRL+C\')
def callback(ch, method, properties, body):
    print(" [x] %r" % body)
channel.basic_consume(callback,
                      queue=queue_name,
                      no_ack=True)
channel.start_consuming()

10、binding

    rabbitmq通过binding将exchange与queue关联起来，这样rabbitmq就知道如何正确将消息路由到指定的queue了。

11、binding key

    绑定exchange与queue的同时，一般会指定binding key；binding key 与routing key匹配时，消息会被路由到相应的queue。

12、exchange type

    rabbitmq常用的exchange type有fanout、direct、topic、headers四种。

13、fanout

    fanout类型的exchange 路由规则非常简单，他会把所有发送到该exchange的消息路由到所有与他绑定的queue中

14、direct

direct会把消息路由到那些binding key 与routing key 完全匹配的queue中

emit_log_direct.py

         
         
#!/usr/bin/env python
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(
        host=\'localhost\'))
channel = connection.channel()
channel.exchange_declare(exchange=\'direct_logs\',
                         type=\'direct\')
severity = sys.argv[1] if len(sys.argv) > 1 else \'info\'
message = \' \'.join(sys.argv[2:]) or \'Hello World!\'
channel.basic_publish(exchange=\'direct_logs\',
                      routing_key=severity,
                      body=message)
print(" [x] Sent %r:%r" % (severity, message))
connection.close()

recieve_logs_direct.py

          
          
#!/usr/bin/env python
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(
        host=\'localhost\'))
channel = connection.channel()
channel.exchange_declare(exchange=\'direct_logs\',
                         type=\'direct\')
result = channel.queue_declare(exclusive=True)
queue_name = result.method.queue
severities = sys.argv[1:]
if not severities:
    sys.stderr.write("Usage: %s [info] [warning] [error]\\n" % sys.argv[0])
    sys.exit(1)
for severity in severities:
    channel.queue_bind(exchange=\'direct_logs\',
                       queue=queue_name,
                       routing_key=severity)
print(\' [*] Waiting for logs. To exit press CTRL+C\')
def callback(ch, method, properties, body):
    print(" [x] %r:%r" % (method.routing_key, body))
channel.basic_consume(callback,
                      queue=queue_name,
                      no_ack=True)
channel.start_consuming()

15、topic

    前面讲到direct类型的exchange路由规则完全匹配binding key 与routing key，但这种严格的匹配方法在很多情况下不能满足实际需求，可能需要模糊匹配的情况，因此通过topic类型进行模糊匹配

以上图中的配置为例，routingKey=”quick.orange.rabbit”的消息会同时路由到Q1与Q2，routingKey=”lazy.orange.fox”的消息会路由到Q1，routingKey=”lazy.brown.fox”的消息会路由到Q2，routingKey=”lazy.pink.rabbit”的消息会路由到Q2（只会投递给Q2一次，虽然这个routingKey与Q2的两个bindingKey都匹配）；routingKey=”quick.brown.fox”、routingKey=”orange”、routingKey=”quick.orange.male.rabbit”的消息将会被丢弃，因为它们没有匹配任何bindingKey。

emit_log_topic.py

         
         
#!/usr/bin/env python
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(
        host=\'localhost\'))
channel = connection.channel()
channel.exchange_declare(exchange=\'topic_logs\',
                         type=\'topic\')
routing_key = sys.argv[1] if len(sys.argv) > 1 else \'anonymous.info\'
message = \' \'.join(sys.argv[2:]) or \'Hello World!\'
channel.basic_publish(exchange=\'topic_logs\',
                      routing_key=routing_key,
                      body=message)
print(" [x] Sent %r:%r" % (routing_key, message))
connection.close()

client.py

         
         
#!/usr/bin/env python
import pika
import sys
connection = pika.BlockingConnection(pika.ConnectionParameters(
        host=\'localhost\'))
channel = connection.channel()
channel.exchange_declare(exchange=\'topic_logs\',
                         type=\'topic\')
result = channel.queue_declare(exclusive=True)
queue_name = result.method.queue
binding_keys = sys.argv[1:]
if not binding_keys:
    sys.stderr.write("Usage: %s [binding_key]...\\n" % sys.argv[0])
    sys.exit(1)
for binding_key in binding_keys:
    channel.queue_bind(exchange=\'topic_logs\',
                       queue=queue_name,
                       routing_key=binding_key)
print(\' [*] Waiting for logs. To exit press CTRL+C\')
def callback(ch, method, properties, body):
    print(" [x] %r:%r" % (method.routing_key, body))
channel.basic_consume(callback,
                      queue=queue_name,
                      no_ack=True)
channel.start_consuming()

16、headers

    header是根据消息内容中的header属性进行匹配的。

17、RPC

    

• When the Client starts up, it creates an anonymous exclusive callback queue.
• For an RPC request, the Client sends a message with two properties: reply_to, which is set to the callback queue and correlation_id, which is set to a unique value for every request.
• The request is sent to an rpc_queue queue.
• The RPC worker (aka: server) is waiting for requests on that queue. When a request appears, it does the job and sends a message with the result back to the Client, using the queue from thereply_to field.
• The client waits for data on the callback queue. When a message appears, it checks thecorrelation_id property. If it matches the value from the request it returns the response to the application.

rpc_server.py

        
        
#!/usr/bin/env python
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(
        host=\'localhost\'))
channel = connection.channel()
channel.queue_declare(queue=\'rpc_queue\')
def fib(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fib(n-1) + fib(n-2)
def on_request(ch, method, props, body):
    n = int(body)
    print(" [.] fib(%s)" % n)
    response = fib(n)
    ch.basic_publish(exchange=\'\',
                     routing_key=props.reply_to,
                     properties=pika.BasicProperties(correlation_id = \\
                                                         props.correlation_id),
                     body=str(response))
    ch.basic_ack(delivery_tag = method.delivery_tag)
channel.basic_qos(prefetch_count=1)
channel.basic_consume(on_request, queue=\'rpc_queue\')
print(" [x] Awaiting RPC requests")
channel.start_consuming()

rpc_receiver.py

        
        
#!/usr/bin/env python
import pika
import uuid
class FibonacciRpcClient(object):
    def __init__(self):
        self.connection = pika.BlockingConnection(pika.ConnectionParameters(
                host=\'localhost\'))
        self.channel = self.connection.channel()
        result = self.channel.queue_declare(exclusive=True)
        self.callback_queue = result.method.queue
        self.channel.basic_consume(self.on_response, no_ack=True,
                                   queue=self.callback_queue)
    def on_response(self, ch, method, props, body):
        if self.corr_id == props.correlation_id:
            self.response = body
    def call(self, n):
        self.response = None
        self.corr_id = str(uuid.uuid4())
        self.channel.basic_publish(exchange=\'\',
                                   routing_key=\'rpc_queue\',
                                   properties=pika.BasicProperties(
                                         reply_to = self.callback_queue,
                                         correlation_id = self.corr_id,
                                         ),
                                   body=str(n))
        while self.response is None:
            self.connection.process_data_events()
        return int(self.response)
fibonacci_rpc = FibonacciRpcClient()
print(" [x] Requesting fib(30)")
response = fibonacci_rpc.call(30)
print(" [.] Got %r" % response)

附录：

    自己写的一些实验程序

producer.py

       
       
#coding=utf-8
import sys
import pika
import time
import random
"""
在数据处理结束后发送ack，这样RabbitMQ Server会认为Message Deliver 成功。
持久化queue，可以防止RabbitMQ Server 重启或者crash引起的数据丢失。
持久化Message，理由同上。
答案是否定的。问题就在与RabbitMQ需要时间去把这些信息存到磁盘上，这个time window虽然短，但是它的确还是有。在这个时间窗口内如果数据没有保存，数据还会丢失。还有另一个原因就是RabbitMQ并不是为每个Message都做fsync：它可能仅仅是把它保存到Cache里，还没来得及保存到物理磁盘上。
因此这个持久化还是有问题。但是对于大多数应用来说，这已经足够了。当然为了保持一致性，你可以把每次的publish放到一个transaction中。这个transaction的实现需要user defined codes。
那么商业系统会做什么呢？一种可能的方案是在系统panic时或者异常重启时或者断电时，应该给各个应用留出时间去flash cache，保证每个应用都能exit gracefully。
"""
class RbMQProducer(object):
    def __init__(self):
        self.connection = pika.BlockingConnection(pika.ConnectionParameters(host=\'localhost\'))
    def send_message(self):
        channel = self.connection.channel()
        channel.queue_declare(queue=\'hello\')
        for i in xrange(10):
            time.sleep(1)
            message = " ".join(sys.argv[1:]) or "hello world! i am {}".format(i)
            channel.basic_publish(exchange=\'\',
                                  routing_key=\'hello\',
                                  body=message)
            print(" [x] Sent \'Hello World!\'")
    def send_message_durable(self):
        channel = self.connection.channel()
        channel.queue_declare(queue=\'hello\',durable=True)
        for i in xrange(10):
            time.sleep(1)
            message = " ".join(sys.argv[1:]) or "hello world! i am {}".format(i)
            channel.basic_publish(exchange=\'\',
                                  routing_key="hello",
                                  body=message,
                                  properties=pika.BasicProperties(
                                      delivery_mode=2,    #make the message persistent
                                  ))
    def exchange_message(self):
        channel = self.connection.channel()
        channel.exchange_declare(exchange=\'logs\',type=\'fanout\')
        for i in xrange(10):
            message = " ".join(sys.argv[1:]) or "exchange info : Hello world {}!".format(i)
            channel.basic_publish(exchange=\'logs\',
                                  routing_key=\'\',
                                  body=message)
            print " [{}] send {}".format(i,message)
            time.sleep(1.5)
    def direct_message(self):
        channel = self.connection.channel()
        channel.exchange_declare(exchange=\'direct_logs\',
                                 type=\'direct\')
        severity = sys.argv[1] if len(sys.argv)>1 else [\'info\',\'warning\',\'error\']
        for i in xrange(10):
            message = " ".join(sys.argv[2:]) or \'hello world!\'
            routing_key_random = severity[random.randint(0,len(severity)-1)]
            channel.basic_publish(exchange=\'direct_logs\',
                                  routing_key=routing_key_random,
                                  body=message)
            print "[{}]: {}".format(routing_key_random,message)
            time.sleep(1.5)
    def topic_message(self):
        channel = self.connection.channel()
        channel.exchange_declare(exchange=\'topic_logs\',
                                 type=\'topic\')
        routing_key = sys.argv[1] if len(sys.argv)>1 else ["anonymous.info","lck.info","lck.error"]
        for i in xrange(10):
            routing_key_value = routing_key[random.randint(0,len(routing_key)-1)]
            message = " ".join(sys.argv[2:]) or "hello world! "
            channel.basic_publish(exchange=\'topic_logs\',
                                  routing_key=routing_key_value,
                                  body=message)
            print "[{}] : {}".format(routing_key_value,message)
            time.sleep(1.5)
    def rpc_message(self):
        channel = self.connection.channel()
        channel.queue_declare(queue=\'rpc_queue\')
        channel.basic_qos(prefetch_count=1)
        channel.basic_consume(self.on_request,queue=\'rpc_queue\')
        print "[x] Awaiting rpc requests"
        channel.start_consuming()
    def on_request(self,ch, method, props, body):
        n = int(body)
        response = random.randint(100,1000)
        print "[.]{}".format(response)
        ch.basic_publish(exchange=\'\',
                         routing_key=props.reply_to,
                         properties=pika.BasicProperties(correlation_id=props.correlation_id),
                         body=str(response))
        ch.basic_ack(delivery_tag=method.delivery_tag)
    def __del__(self):
        self.connection.close()
if __name__ == "__main__":
    rm_producer = RbMQProducer()
    # rm_producer.send_message()
    # rm_producer.send_message_durable()
    rm_producer.exchange_message()
    # rm_producer.direct_message()
    # rm_producer.topic_message()
    # rm_producer.rpc_message()

receiver.py

      
      
#coding=utf-8
import sys
import pika
import time
import uuid
class RbMQconsume(object):
    def __init__(self):
        self.connection = pika.BlockingConnection(pika.ConnectionParameters(host=\'localhost\'))
    def callback(self, ch, method, properties, body):
        print(" [x] Received %r" % body)
        time.sleep( body.count(\'.\'))
        print(" [x] Done")
    def receive_message(self):
        channel = self.connection.channel()
        channel.queue_declare(queue=\'hello\')
        channel.basic_consume(self.callback,
                              queue=\'hello\',
                              no_ack=True)
        print(\' [*] Waiting for messages. To exit press CTRL+C\')
        channel.start_consuming()
    def safe_callback(self,ch,method,properties,body):
        print " [x] Received %r"%(body)
        time.sleep(body.count(\'.\'))
        print( " [x] done !")
        ch.basic_ack(delivery_tag=method.delivery_tag)
    def safe_receive_message(self):
        channel = self.connection.channel()
        channel.queue_declare(queue=\'hello\',durable=True)
        channel.basic_qos(prefetch_count=1)
        channel.basic_consume(self.safe_callback,
                              queue=\'hello\')
        print(\' [*] Waiting for messages. To exit press CTRL+C\')
        channel.start_consuming()
    #exchange方法
    def exchange_callback(self,ch,method,properties,body):
        print " [ x ] {}".format(body)
    def receieve_exchange_message(self):
        channel = self.connection.channel()
        channel.exchange_declare(exchange=\'logs\',
                                 type=\'fanout\')
        result = channel.queue_declare(exclusive=True)
        queue_name = result.method.queue
        channel.queue_bind(exchange=\'logs\',queue=queue_name)
        channel.basic_consume(self.exchange_callback,queue=queue_name,no_ack=True)
        channel.start_consuming()
        print(\' [*] Waiting for messages. To exit press CTRL+C\')
    #direct 方法
    def direct_callback(self, ch, method, properties, body):
        print " [ {} ] {}".format(method.routing_key,body)
    def receieve_direct_message(self):
        channel = self.connection.channel()
        channel.exchange_declare(exchange=\'direct_logs\',
                                 type =\'direct\')
        result = channel.queue_declare(exclusive=True)
        queue_name = result.method.queue
        severities = [\'warning\', \'error\']
        for severity in severities:
            channel.queue_bind(exchange=\'direct_logs\',
                               queue=queue_name,
                               routing_key=severity)
        channel.basic_consume(self.direct_callback,
                              queue=queue_name,
                              no_ack=True)
        channel.start_consuming()
    #topic
    def topic_callback(self, ch, method, properties, body):
        print " [ {} ] {}".format(method.routing_key, body)
    def receieve_topic_message(self):
        channel = self.connection.channel()
        channel.exchange_declare(exchange=\'topic_logs\',
                                 type=\'topic\')
        result = channel.queue_declare(exclusive=True)
        queue_name = result.method.queue
        binding_keys = ["anonymous.info","lck.info","lck.error"]
        binding_key = \'lck.*\'
        channel.queue_bind(exchange=\'topic_logs\',
                           queue=queue_name,
                           routing_key=binding_key)
        channel.basic_consume(self.topic_callback,
                              queue= queue_name,
                              no_ack=True)
        channel.start_consuming()
    #rpc
    def rpc_callback(self,ch,method,props,body):
        if self.corr_id == props.correlation_id:
            self.response = body
    def receieve_rpc_message(self,n):
        channel = self.connection.channel()
        result = channel.queue_declare(exclusive=True)
        callback_queue = result.method.queue
        channel.basic_consume(self.rpc_callback,
                              no_ack=True,
                              queue=callback_queue)
        self.response = None
        self.corr_id = str(uuid.uuid4())
        channel.basic_publish(exchange=\'\',
                              routing_key=\'rpc_queue\',
                              properties=pika.BasicProperties(reply_to=callback_queue,correlation_id=self.corr_id),
                              body=str(n))
        while self.response is None:
            self.connection.process_data_events()
        return int(self.response)
    def __del__(self):
        self.connection.close()
if  __name__=="__main__":
    rqconsume = RbMQconsume()
    # rqconsume.safe_receive_message()
    rqconsume.receieve_exchange_message()
    # rqconsume.receieve_direct_message()
    # rqconsume.receieve_topic_message()
    # print rqconsume.receieve_rpc_message(30)