rocketmq 雪崩后客户端超时

rocketmq 雪崩后客户端超时

看了源代码,mq broker 方核心日志信息:

1. 通过 mqAdmin 的消息 id,查看信息的 born 和 stroe 时间

2. 通过慢查日志查看.

defaultMessageStore.java 中

 PutMessageResult putMessage(MessageExtBrokerInner msg) {

log. warn( "putMessage not in lock eclipse time(ms)={}, bodyLength={}", eclipseTime, msg.getBody(). length);

3.业务方的统计日志.

4. 线程堆栈统计.

报错异常信息:

com.paygateway.core.service.integration.bean.PayGatewayPaySuccess@462ba599,Flag=644560335968fcbd9e27515f3bc2ac86
com.alibaba.rocketmq.client.exception.MQClientException: Send [2] times, still failed, cost [6002]ms, Topic: PAY_GATEWAY_CALLBACK_SUCCESS_TOPIC, BrokersSent: [broker_cluster_3, broker_cluster_0, null]
For more information, please visit the url, https://github.com/alibaba/RocketMQ/issues/64
        at com.alibaba.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:590)
        at com.alibaba.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:1026)
        at com.alibaba.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:122)


Caused by: com.alibaba.rocketmq.remoting.exception.RemotingTimeoutException: wait response on the channel <100.69.87.39:10911> timeout, 3000(ms)
        at com.alibaba.rocketmq.remoting.netty.NettyRemotingAbstract.invokeSyncImpl(NettyRemotingAbstract.java:378)
        at com.alibaba.rocketmq.remoting.netty.NettyRemotingClient.invokeSync(NettyRemotingClient.java:625)
        at com.alibaba.rocketmq.client.impl.MQClientAPIImpl.sendMessageSync(MQClientAPIImpl.java:395)
        at com.alibaba.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:378)
        at com.alibaba.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendKernelImpl(DefaultMQProducerImpl.java:698)

        at com.alibaba.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:514)

    既要看 provider 也要看 consumer

    业务方报错:

    provider  :

TOPIC_PUT_SIZE

  less  stats-2017-07-14.0.log| grep '2017-07-14 22:27' | grep TOPIC_PUT_SIZE | awk -F' - |\\[|\\]|SUM:|TPS:|AVGPT:' '{ minute=substr($1,12,5); siz[$5]+=$7; tps[$5]+=$8; tpss+=$8} END { print tpss }'
528089


TPS put num

 $  less  ../stats.log| grep '2017-07-15 21:18' | grep TOPIC_PUT_NUMS | awk -F' - |\\[|\\]|SUM:|TPS:|AVGPT:' '{ minute=substr($1,12,5); siz[$5]+=$7; tps[$5]+=$8; tpss+=$8} END { print tpss }'
298.66
 $   less stats-2017-07-14.0.log
$  less  stats-2017-07-14.0.log| grep '2017-07-14 22:27' | grep TOPIC_PUT_NUMS | awk -F' - |\\[|\\]|SUM:|TPS:|AVGPT:' '{ minute=substr($1,12,5); siz[$5]+=$7; tps[$5]+=$8; tpss+=$8} END { print tpss }'
596.28

1000B= 1k /4=256个字符.
22:27分 正常,周五比周四大 (和周四比并没有查很多.)
 $   less stats-2017-07-14.0.log| grep '2017-07-14 22:27' | awk -F' - |\\[|\\]|SUM:|TPS:|AVGPT:' '{ minute=substr($1,12,5); siz[$5]+=$7; tps[$5]+=$8} END { for ( i in tps) print i,tps[i]/10000,siz[i]/10000}' |  sort -n -k 3 -r | head -n 2
TCP_MSG_TOPIC@tcpMsgConsumerGroup 11.7738 706.43
TCP_MSG_HEARTBEATS_TOPIC@lbsGroup 9.46656  567.994

[]$   less stats-2017-07-13.0.log| grep '2017-07-13 22:27' | awk -F' - |\\[|\\]|SUM:|TPS:|AVGPT:' '{ minute=substr($1,12,5); siz[$5]+=$7; tps[$5]+=$8} END { for ( i in tps) print i,tps[i]/10000,siz[i]/10000}' |  sort -n -k 3 -r | head -n 2
TCP_MSG_TOPIC@tcpMsgConsumerGroup 10.2319 613.914
TCP_MSG_HEARTBEATS_TOPIC@lbsGroup 7.92051 475.231
22:29 分.异常 周五比周四小.
[]$   less stats-2017-07-14.0.log| grep '2017-07-14 22:29' | awk -F' - |\\[|\\]|SUM:|TPS:|AVGPT:' '{ minute=substr($1,12,5); siz[$5]+=$7; tps[$5]+=$8} END { for ( i in tps) print i,tps[i]/10000,siz[i]/10000}' |  sort -n -k 3 -r | head -n 2
TCP_MSG_TOPIC@tcpMsgConsumerGroup 7.05011 423.006
TCP_MSG_HEARTBEATS_TOPIC@lbsGroup 6.04157 362.494
[]$   less stats-2017-07-13.0.log| grep '2017-07-13 22:29' | awk -F' - |\\[|\\]|SUM:|TPS:|AVGPT:' '{ minute=substr($1,12,5); siz[$5]+=$7; tps[$5]+=$8} END { for ( i in tps) print i,tps[i]/10000,siz[i]/10000}' |  sort -n -k 3 -r | head -n 2
TCP_MSG_TOPIC@tcpMsgConsumerGroup 10.1158 606.95
TCP_MSG_HEARTBEATS_TOPIC@lbsGroup 7.75625 465.375





 21:29 正常 周五比周四的大
[]$   less stats-2017-07-14.0.log| grep '2017-07-14 21:29' | awk -F' - |\\[|\\]|SUM:|TPS:|AVGPT:' '{ minute=substr($1,12,5); siz[$5]+=$7; tps[$5]+=$8} END { for ( i in tps) print i,tps[i]/10000,siz[i]/10000}' |  sort -n -k 2 -r | head -n 2
TCP_MSG_TOPIC@tcpMsgConsumerGroup 15.0323 901.939
TCP_MSG_SMDATA_TOPIC@lbsGroup 10.3313 619.879
[]$   less stats-2017-07-13.0.log| grep '2017-07-13 21:29' | awk -F' - |\\[|\\]|SUM:|TPS:|AVGPT:' '{ minute=substr($1,12,5); siz[$5]+=$7; tps[$5]+=$8} END { for ( i in tps) print i,tps[i]/10000,siz[i]/10000}' |  sort -n -k 2 -r | head -n 2
TCP_MSG_TOPIC@tcpMsgConsumerGroup 13.3589 801.534
TCP_MSG_SMDATA_TOPIC@lbsGroup 8.98617 539.17