Python使用rabbitmq作爬虫的消息队列的一个例子

先来一张图,自己画的:

使用consumer自己即是消费者也是生产者,可以开多线程,产生多个consumer,既消费又产出,producter的消息应该是传到Ex中,方便说明问题,直接搭在Queue上了

代码:

只需要写callback里面的函数,以参数形式传到recv_task中就可以了
这里callback注意base_ack,及时通知rabbitmq,清理已经完成的任务,否则会造成队列堵塞.consumer中出现异常也会造成阻塞哦,问题会一大堆呢

import pika
def send_task(queue_name, task):
    connection = pika.BlockingConnection(pika.ConnectionParameters(
        'localhost'))
    channel = connection.channel()
    # 声明一个exchange
    channel.exchange_declare(exchange='messages', exchange_type='direct')
    # 声明queue
    channel.queue_declare(queue=queue_name, durable=True)
    # n RabbitMQ a message can never be sent directly to the queue, it always needs to go through an exchange.
    channel.queue_bind(exchange='messages',
                       queue=queue_name,
                       routing_key=queue_name)

    channel.basic_publish(exchange='messages',
                          routing_key=queue_name,
                          body=task,
                          properties=pika.BasicProperties(
                              delivery_mode=2,  # make message persistent
                          )
                          )
    print(" [x] Sent %r" % task)
    connection.close()

def recv_task(queue_name, callback):
    connection = pika.BlockingConnection(pika.ConnectionParameters(
        'localhost'))
    channel = connection.channel()
    # 声明一个exchange
    channel.exchange_declare(exchange='messages', exchange_type='direct')
    # 声明queue
    channel.queue_declare(queue=queue_name, durable=True)
    # n Rabb