Consumer启动后会请求broker,broker的PullMessageProcessor的processRequest会对请求进行处理
主要步骤
1.判断确保订阅组是否存在
2.判断订阅组是否可以消费消息
3. 检查 topic 是否存在
4. 检查 topic 权限
5.检查队列有效性
6.判断消费者组是否存在
7.广播模式下判断是否允许广播方式消费
8. 判断 SubscriptionData( 保存 topoc , tag 等信息 ) 是否存在
9. 获取最大位置 maxOffsetPy=fileFromOffset+ 当前文件写到的位置 ( 即为 PHYSICALOFFSET)
10. 通过 topic 和 queueId 获取 consumeQueue
11. 通过 Consumer 传过来的 offset 获取 SelectMapedBufferResult( 封装了 bytebuffer)
12. 从 bytebuffer 中获取 CommitLog offset , size ,和 tagsCode
13.过滤消息
14. 通过 commitLogOffset 和 size 获取消息所在文件的 mapedFile
15. 调用 mapedFile 的 selectMapedBuffer 方法获取 byteBuffer 并封装成
SelectMapedBufferResult
16.计算下次请求的位置
17. 把状态,下次请求的位置,当前队列最大的 offset 和最小的 offset 和消息对应的 bytebuffer 封装成对象返回
18.根据topic和group从offsetTable中获取ConcurrentHashMap(key为队列id,value为消费进度),将消费进度设置到map中
详细过程
pullMessageProcessor的processRequest负责处理consumer的消费请求,broker会根据offset去获取消息
final GetMessageResult getMessageResult =
this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), requestHeader.getTopic(),
requestHeader.getQueueId(), requestHeader.getQueueOffset(), requestHeader.getMaxMsgNums(), subscriptionData);
这前面都是一些校验,获取消息和处理消息一样都是先进入DefaultMessageStore(处理消息是putMessage,获取消息是getMessage)
进来之后会先通过topic和queueId获取对应的ConsumeQueue,有了ConsumeQueue就可以操作逻辑队列
ConsumeQueue consumeQueue = findConsumeQueue(topic, queueId);
public ConsumeQueue findConsumeQueue(String topic, int queueId) {
ConcurrentHashMap<Integer, ConsumeQueue> map = consumeQueueTable.get(topic);
if (null == map) {
ConcurrentHashMap<Integer, ConsumeQueue> newMap = new ConcurrentHashMap<Integer, ConsumeQueue>(128);
ConcurrentHashMap<Integer, ConsumeQueue> oldMap = consumeQueueTable.putIfAbsent(topic, newMap);
if (oldMap != null) {
map = oldMap;
} else {
map = newMap;
}
}
ConsumeQueue logic = map.get(queueId);
if (null == logic) {
ConsumeQueue newLogic = new ConsumeQueue(//
topic, //
queueId, //
StorePathConfigHelper.getStorePathConsumeQueue(this.messageStoreConfig.getStorePathRootDir()), //
this.getMessageStoreConfig().getMapedFileSizeConsumeQueue(), //
this);
ConsumeQueue oldLogic = map.putIfAbsent(queueId, newLogic);
if (oldLogic != null) {
logic = oldLogic;
} else {
logic = newLogic;
}
}
return logic;
}
获取到ConsumeQueue之后,会判断offset的合理性
minOffset = consumeQueue.getMinOffsetInQuque();
maxOffset = consumeQueue.getMaxOffsetInQuque();
if (maxOffset == 0) {
status = GetMessageStatus.NO_MESSAGE_IN_QUEUE;
nextBeginOffset = nextOffsetCorrection(offset, 0);
} else if (offset < minOffset) {
status = GetMessageStatus.OFFSET_TOO_SMALL;
nextBeginOffset = nextOffsetCorrection(offset, minOffset);
} else if (offset == maxOffset) {
status = GetMessageStatus.OFFSET_OVERFLOW_ONE;
nextBeginOffset = nextOffsetCorrection(offset, offset);
} else if (offset > maxOffset) {
status = GetMessageStatus.OFFSET_OVERFLOW_BADLY;
if (0 == minOffset) {
nextBeginOffset = nextOffsetCorrection(offset, minOffset);
} else {
nextBeginOffset = nextOffsetCorrection(offset, maxOffset);
}
} else {
//消费位置合法
}
消费的位置合法,那么就从consumeQueue中获取SelectMapedBufferResult(封装了bytebuffer)
SelectMapedBufferResult bufferConsumeQueue = consumeQueue.getIndexBuffer(offset);
public SelectMapedBufferResult getIndexBuffer(final long startIndex) {
int mapedFileSize = this.mapedFileSize;
long offset = startIndex * CQStoreUnitSize;
if (offset >= this.getMinLogicOffset()) {
MapedFile mapedFile = this.mapedFileQueue.findMapedFileByOffset(offset);
if (mapedFile != null) {
SelectMapedBufferResult result = mapedFile.selectMapedBuffer((int) (offset % mapedFileSize));
return result;
}
}
return null;
}
逻辑队列中一个数据的大小是20,所以真正的offset需要乘以20,然后获取对应的MapedFile
findMapedFileByOffset就是根据offset来获取对应的MapedFile
public MapedFile findMapedFileByOffset(final long offset, final boolean returnFirstOnNotFound) {
try {
this.readWriteLock.readLock().lock();
MapedFile mapedFile = this.getFirstMapedFile();
if (mapedFile != null) {
int index =
(int) ((offset / this.mapedFileSize) - (mapedFile.getFileFromOffset() / this.mapedFileSize));
if (index < 0 || index >= this.mapedFiles.size()) {
//日志打印....
}
try {
return this.mapedFiles.get(index);
} catch (Exception e) {
if (returnFirstOnNotFound) {
return mapedFile;
}
}
}
} catch (Exception e) {
log.error("findMapedFileByOffset Exception", e);
} finally {
this.readWriteLock.readLock().unlock();
}
return null;
}
假如文件大小为1024,现在有3个文件,即mapedFiles.size()==3,offset为3406
第一个文件 第二个文件 第三个文件 0~1023 1024~2047 2048~3071 那么,offset为3046即为第3个文件,以下式子可求
Index = (int)(offset/fileSize)=(int)(3406/2014)=2
由于这种情况下mapedFile.getFileFromOffset()为0,那么可省略不计
另外的情况:
如果第一个文件被删除了,那么就需要后面的式子
第一个文件 第二个文件 第三个文件 1024~2047
2048~3071 3072~4096
Offset在第2个文件,index为1
而根据Index = (int)(offset/fileSize)=(int)(3406/2014)=2
所以需要减去mapedFile.getFileFromOffset() / this.mapedFileSize
获取到MapedFile之后,就可以使用bytebuffer来读取了
public SelectMapedBufferResult selectMapedBuffer(int pos) {
if (pos < this.wrotePostion.get() && pos >= 0) {
if (this.hold()) {
ByteBuffer byteBuffer = this.mappedByteBuffer.slice();
byteBuffer.position(pos);
int size = this.wrotePostion.get() - pos;
ByteBuffer byteBufferNew = byteBuffer.slice();
byteBufferNew.limit(size);
return new SelectMapedBufferResult(this.fileFromOffset + pos, byteBufferNew, size, this);
}
}
return null;
}
pos=(int) (offset % mapedFileSize)=(int)(3406%1024)=334
size=当前消息写到的位置-334=可以读取的消息的大小
最后封装成SelectMapedBufferResult返回
获取到SelectMapedBufferResult之后,就可以读取消息了,代码中是一个循环
if (bufferConsumeQueue != null) {
try {
status = GetMessageStatus.NO_MATCHED_MESSAGE;
long nextPhyFileStartOffset = Long.MIN_VALUE;
long maxPhyOffsetPulling = 0;
int i = 0;
final int MaxFilterMessageCount = 16000;
final boolean diskFallRecorded = this.messageStoreConfig.isDiskFallRecorded();
for (; i < bufferConsumeQueue.getSize() && i < MaxFilterMessageCount; i += ConsumeQueue.CQStoreUnitSize) {
long offsetPy = bufferConsumeQueue.getByteBuffer().getLong();
int sizePy = bufferConsumeQueue.getByteBuffer().getInt();
long tagsCode = bufferConsumeQueue.getByteBuffer().getLong();
maxPhyOffsetPulling = offsetPy;
// 说明物理文件正在被删除
if (nextPhyFileStartOffset != Long.MIN_VALUE) {
if (offsetPy < nextPhyFileStartOffset)
continue;
}
// 判断是否拉磁盘数据
boolean isInDisk = checkInDiskByCommitOffset(offsetPy, maxOffsetPy);
// 此批消息达到上限了
if (this.isTheBatchFull(sizePy, maxMsgNums, getResult.getBufferTotalSize(), getResult.getMessageCount(),
isInDisk)) {
break;
}
// 消息过滤
if (this.messageFilter.isMessageMatched(subscriptionData, tagsCode)) {
SelectMapedBufferResult selectResult = this.commitLog.getMessage(offsetPy, sizePy);
if (selectResult != null) {
this.storeStatsService.getGetMessageTransferedMsgCount().incrementAndGet();
getResult.addMessage(selectResult);
status = GetMessageStatus.FOUND;
nextPhyFileStartOffset = Long.MIN_VALUE;
} else {
if (getResult.getBufferTotalSize() == 0) {
status = GetMessageStatus.MESSAGE_WAS_REMOVING;
}
// 物理文件正在被删除,尝试跳过
nextPhyFileStartOffset = this.commitLog.rollNextFile(offsetPy);
}
} else {
if (getResult.getBufferTotalSize() == 0) {
status = GetMessageStatus.NO_MATCHED_MESSAGE;
}
if (log.isDebugEnabled()) {
log.debug("message type not matched, client: " + subscriptionData + " server: " + tagsCode);
}
}
}
if (diskFallRecorded) {
long fallBehind = maxOffsetPy - maxPhyOffsetPulling;
brokerStatsManager.recordDiskFallBehindSize(group, topic, queueId, fallBehind);
}
nextBeginOffset = offset + (i / ConsumeQueue.CQStoreUnitSize);
long diff = maxOffsetPy - maxPhyOffsetPulling;
long memory = (long) (StoreUtil.TotalPhysicalMemorySize
* (this.messageStoreConfig.getAccessMessageInMemoryMaxRatio() / 100.0));
getResult.setSuggestPullingFromSlave(diff > memory);
} finally {
bufferConsumeQueue.release();
}
} else {
status = GetMessageStatus.OFFSET_FOUND_NULL;
nextBeginOffset = nextOffsetCorrection(offset, consumeQueue.rollNextFile(offset));
log.warn("consumer request topic: " + topic + "offset: " + offset + " minOffset: " + minOffset + " maxOffset: "
+ maxOffset + ", but access logic queue failed.");
}
}
将 SelectMapedBufferResult 添加到 GetMessageResult 中
逻辑队列是大小为20,每次+20
我们在上面已经获取到可以读取的消息的大小为334,那么就可以读取334/20个消息,MaxFilterMessageCount=16000,最大可读取的消息大小
根据consumequeue的结构,可以知道由3部分组成,那么依次取出来commitlog offset,size,tag
然后根据拿tag和SubscriptionData进行比对,进行过滤消息
消息匹配之后,有了offset和size,则可以调用commitlog的getMessage方法获取SelectMapedBufferResult(获取过程和上面说的getIndexBuffer类似),然后
中,在PullRequestHoldService的线程中会每 10 秒检查 ConsumeQueue 最大 offset 是否大于 Request 中的 offSet ,如果大于则有消息产生,那么执行 PullMessageProcessor.processRequest 方法(该线程的作用在后面的章节会讲到)
退出循环后,会计算下次的offset,放到GetMessageResult中返回
当pullMessageProcessor获取到GetMessageResult,然后将消息体,下次访问的offset等设置到response中返回
当状态为PULL_NOT_FOUND的时候,需要将RequestCommand,channel,offset,SubscriptionData等信息组装成PullRequest对象,然后放到PullRequestHoldService
case ResponseCode.PULL_NOT_FOUND:
if (brokerAllowSuspend && hasSuspendFlag) {
long pollingTimeMills = suspendTimeoutMillisLong;
if (!this.brokerController.getBrokerConfig().isLongPollingEnable()) {
pollingTimeMills = this.brokerController.getBrokerConfig().getShortPollingTimeMills();
}
String topic = requestHeader.getTopic();
long offset = requestHeader.getQueueOffset();
int queueId = requestHeader.getQueueId();
PullRequest pullRequest = new PullRequest(request, channel, pollingTimeMills,
this.brokerController.getMessageStore().now(), offset, subscriptionData);
this.brokerController.getPullRequestHoldService().suspendPullRequest(topic, queueId, pullRequest);//放到PullRequestHoldService的pullRequestTable中
response = null;
break;
}