rabbitmq--prefetch count

消费者在开启acknowledge的情况下,对接收到的消息可以根据业务的需要异步对消息进行确认。

然而在实际使用过程中,由于消费者自身处理能力有限,从rabbitmq获取一定数量的消息后,希望rabbitmq不再将队列中的消息推送过来,当对消息处理完后(即对消息进行了ack,并且有能力处理更多的消息)再接收来自队列的消息。在这种场景下,我们可以通过设置basic.qos信令中的prefetch_count来达到这种效果。

先直观的看看设置了prefetch_count的效果,:

1) 对比测试:两个消费者都订阅同一队列,no_ack均设置为false即开启acknowledge机制,且均未设置prefetch_count,向队列发布5条消息

结果:不管消息是否被ack,rabbitmq会轮流向两个消费者投递消息,第一个消费者收到"1","3","5"三条消息, 第二个消费者收到"2","4"两条消息。



2)prefetch_count设置测试:两个消费者都订阅同一队列,开启acknowledge机制,第一个消费者prefetch_count设置为1,另一个消费者未设置prefetch_count,同样向队列发布5条消息

结果:rabbitmq向第一个消费者投递了一条消息后,消费者未对该消息进行ack,rabbitmq不会再向该消费者投递消息,剩下的四条消息均投递给了第二个消费者



看完效果后,再来看看rabbitmq里的一些实现。

1. rabbitmq对basic.qos信令的处理

首先,basic.qos是针对channel进行设置的,也就是说只有在channel建立之后才能发送basic.qos信令。

在rabbitmq的实现中,每个channel都对应会有一个rabbit_limiter进程,当收到basic.qos信令后,在rabbit_limiter进程中记录信令中prefetch_count的值,同时记录的还有该channel未ack的消息个数。

注:其实basic.qos里还有另外两个参数可进行设置(global和prefetch_size),但rabbitmq没有相应的实现。

2. 队列中的消息投递给消费者时的处理

当rabbitmq要将队列中的一条消息投递给消费者时,会遍历该队列上的消费者列表,选一个合适的消费者,然后将消息投递出去。其中挑选消费者的一个依据就是看消费者对应的channel上未ack的消息数是否达到设置的prefetch_count个数,如果未ack的消息数达到了prefetch_count的个数,则不符合要求。当挑选到合适的消费者后,中断后续的遍历。

rabbit_amqqueue_process.erl

deliver_msgs_to_consumers(_DeliverFun, true, State) ->
    {true, State};
deliver_msgs_to_consumers(DeliverFun, false,
                          State = #q{active_consumers =
                                     ActiveConsumers}) ->
    case priority_queue:out_p(ActiveConsumers) of
        {empty, _} ->
            {false, State};
        {{value, QEntry, Priority}, Tail} ->
            {Stop, State1} =
                deliver_msg_to_consumer(DeliverFun, QEntry,
                                        Priority,
                                        State#q{active_consumers =
                                                Tail}),
            %%如果处理结果为false,遍历下一个消费者
            deliver_msgs_to_consumers(DeliverFun, Stop, State1)
    end.

deliver_msg_to_consumer(DeliverFun, E = {ChPid, Consumer},
                        Priority, State) ->
    ...
    %%判断是否可以将消息投递给该消费者
    case rabbit_limiter:can_send(C#cr.limiter,
                                 Consumer#consumer.ack_required,
                                 Consumer#consumer.tag) of
        %%可以投递,再将该消费者放到队列的尾部
        {continue, Limiter} ->
            AC1 = priority_queue:in(E, Priority,
                                    State#q.active_consumers),
            %%将消息投递给消费者
            deliver_msg_to_consumer0(DeliverFun, Consumer,
                                     C#cr{limiter = Limiter},
                                     State#q{active_consumers = AC1})
    ...

rabbit_limiter.erl

handle_call({can_send, QPid, AckRequired}, _From,
            State = #lim{volume = Volume}) ->
    case prefetch_limit_reached(State) of
        %%未ack的消息数达到prefetch_count设置的个数
        true  -> {reply, false, limit_queue(QPid, State)};
        false -> {reply, true,
                  %%消息需要被ACK, volume加1
                  State#lim{volume = if AckRequired -> Volume + 1;
                                        true        -> Volume
                                     end}}
    end

prefetch_limit_reached(#lim{prefetch_count = Limit, 
                            volume = Volume}) ->
    Limit =/= 0 andalso Volume >= Limit.

3. 消费者对消息ack后的处理

当消费者对消息进行ack后,最终会修改该消费者对应channel中未ack的消息数,这样队列又可以将消息投递给该消费者。

rabbit_limiter.erl

handle_cast({ack, Count}, State = #lim{volume = Volume}) ->
    NewVolume = if Volume == 0 -> 0;
                   true        -> Volume - Count
                end,
    {noreply, maybe_notify(State, State#lim{volume = NewVolume})};

4. 扩展

在AMQP协议(0-9-1)中,有这么一段话


对于rabbitmq来说,最后一句话其实说的就是使用了acknowledge机制情况下,使用prefetch_count进行流量控制。另外在实际研究过程中发现还有channel.flow以及basic.credit(应该属于AMQP 1.0协议)可以进行一些控制,这里没有展开,有时间会研究下相应的机制以及可能使用的场景。


Count(*) 和 count(1) 的问题

05-31

为什么我用List result = _dao.find("SELECT COUNT(*) AS count from DrawingnoMap A");不会报错,rn而当我写成 List result = _dao.find("SELECT COUNT(1) AS count from DrawingnoMap A");就会报下面的异常:rnException in thread "Thread-6" java.lang.NoSuchMethodError: org.hibernate.hql.antlr.HqlBaseParser.recover(Lantlr/RecognitionException;Lantlr/collections/impl/BitSet;)Vrnrn at org.hibernate.hql.antlr.HqlBaseParser.aggregate(HqlBaseParser.java:4421)rnrn at org.hibernate.hql.antlr.HqlBaseParser.identPrimary(HqlBaseParser.java:4052)rnrn at org.hibernate.hql.antlr.HqlBaseParser.primaryExpression(HqlBaseParser.java:861)rnrn at org.hibernate.hql.antlr.HqlBaseParser.atom(HqlBaseParser.java:3438)rnrn at org.hibernate.hql.antlr.HqlBaseParser.unaryExpression(HqlBaseParser.java:3216)17:18:07,531 ERROR PARSER[reportError]:33 - line 1:14: unexpected token: 1rnrn at org.hibernate.hql.antlr.HqlBaseParser.multiplyExpression(HqlBaseParser.java:3098)rnrn at org.hibernate.hql.antlr.HqlBaseParser.additiveExpression(HqlBaseParser.java:2818)rnrn at org.hibernate.hql.antlr.HqlBaseParser.concatenation(HqlBaseParser.java:570)rnrn at org.hibernate.hql.antlr.HqlBaseParser.relationalExpression(HqlBaseParser.java:2586)rnrn at org.hibernate.hql.antlr.HqlBaseParser.equalityExpression(HqlBaseParser.java:2449)rnrn at org.hibernate.hql.antlr.HqlBaseParser.negatedExpression(HqlBaseParser.java:2413)rnrn at org.hibernate.hql.antlr.HqlBaseParser.logicalAndExpression(HqlBaseParser.java:2331)rnrn at org.hibernate.hql.antlr.HqlBaseParser.logicalOrExpression(HqlBaseParser.java:2296)rnrn at org.hibernate.hql.antlr.HqlBaseParser.expression(HqlBaseParser.java:2082)rnrn at org.hibernate.hql.antlr.HqlBaseParser.aliasedExpression(HqlBaseParser.java:2249)rnrn at org.hibernate.hql.antlr.HqlBaseParser.selectedPropertiesList(HqlBaseParser.java:1455)rnrn at org.hibernate.hql.antlr.HqlBaseParser.selectClause(HqlBaseParser.java:1365)rnrn at org.hibernate.hql.antlr.HqlBaseParser.selectFrom(HqlBaseParser.java:1106)rnrn at org.hibernate.hql.antlr.HqlBaseParser.queryRule(HqlBaseParser.java:702)rnrn at org.hibernate.hql.antlr.HqlBaseParser.selectStatement(HqlBaseParser.java:296)rnrn at org.hibernate.hql.antlr.HqlBaseParser.statement(HqlBaseParser.java:159)rnrn at org.hibernate.hql.ast.QueryTranslatorImpl.parse(QueryTranslatorImpl.java:248)rnrn at org.hibernate.hql.ast.QueryTranslatorImpl.doCompile(QueryTranslatorImpl.java:157)rnrn at org.hibernate.hql.ast.QueryTranslatorImpl.compile(QueryTranslatorImpl.java:111)rnrn at org.hibernate.engine.query.HQLQueryPlan.(HQLQueryPlan.java:77)rnrn at org.hibernate.engine.query.HQLQueryPlan.(HQLQueryPlan.java:56)rnrn at org.hibernate.engine.query.QueryPlanCache.getHQLQueryPlan(QueryPlanCache.java:72)rnrn at org.hibernate.impl.AbstractSessionImpl.getHQLQueryPlan(AbstractSessionImpl.java:133)rnrn at org.hibernate.impl.AbstractSessionImpl.createQuery(AbstractSessionImpl.java:112)rnrn at org.hibernate.impl.SessionImpl.createQuery(SessionImpl.java:1623)rnrn at org.springframework.orm.hibernate3.HibernateTemplate$29.doInHibernate(HibernateTemplate.java:832)rnrn at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:362)rnrn at org.springframework.orm.hibernate3.HibernateTemplate.find(HibernateTemplate.java:830)rnrn at org.springframework.orm.hibernate3.HibernateTemplate.find(HibernateTemplate.java:822)rnrn at org.ejs.dao.support.GenericDaoSupport.find(Unknown Source)rnrn at com.application.dao.hibernate.HibernateDrawingnoMapDAO.findByDrawingnoCode(HibernateDrawingnoMapDAO.java:34)rnrn at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)rnrn at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)rnrn at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)rnrn at java.lang.reflect.Method.invoke(Method.java:585)rnrn at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:280)rnrn at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:187)rnrn at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:154)rnrn at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:104)rnrn at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)rnrn at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:210)rnrn at $Proxy118.findByDrawingnoCode(Unknown Source)rnrn at com.application.service.pojo.POJODrawingnoMapService.findByDrawingnoCode(POJODrawingnoMapService.java:80)rnrn at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)rnrn at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)rnrn at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)rnrn at java.lang.reflect.Method.invoke(Method.java:585)rnrn at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:280)rnrn at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:187)rnrn at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:154)rnrn at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)rnrn at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)rnrn at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:210)rnrn at $Proxy119.findByDrawingnoCode(Unknown Source)rnrn at com.application.swing.business.DrawingnoMapPlugin.doRefresh(DrawingnoMapPlugin.java:304)rnrn at com.application.swing.business.DrawingnoMapPlugin.doStart(DrawingnoMapPlugin.java:298)rnrn at com.application.swing.MainFrame$1.run(MainFrame.java:176)rnrn at java.lang.Thread.run(Thread.java:595)rn我很纳闷,难道是hibernate 不支持count(1)???rnrn

没有更多推荐了,返回首页

私密
私密原因:
请选择设置私密原因
  • 广告
  • 抄袭
  • 版权
  • 政治
  • 色情
  • 无意义
  • 其他
其他原因:
120
出错啦
系统繁忙,请稍后再试

关闭