Pulsar源码解析-客户端-生产者消息发送底层实现

前几篇介绍过生产者创建的底层实现，本章介绍发送的实现细节

一、分区生产者和非分区生产发送的差异

消息发送存在分区和非分区的Producer代码实现，先介绍一下区别点，然后后面流程都一样。

分区Producer需要实现选择一个具体的Producer实现，怎么选择的呢？

创建生产者时会配置路由策略，看一下实现：

public class PartitionedProducerImpl<T> {
	// 调用是在PartitionedProducerImpl构造中
    private MessageRouter getMessageRouter() {
        MessageRouter messageRouter;
		// 从配置中获取路由模式
        MessageRoutingMode messageRouteMode = conf.getMessageRoutingMode();

        switch (messageRouteMode) {
        	// 自定义实现
            case CustomPartition:
                messageRouter = checkNotNull(conf.getCustomMessageRouter());
                break;
                // 多个分区中随机固定一个
            case SinglePartition:
                messageRouter = new SinglePartitionMessageRouterImpl(
                        ThreadLocalRandom.current().nextInt(topicMetadata.numPartitions()), conf.getHashingScheme());
                break;
                // 轮询
            case RoundRobinPartition:
            default:
                messageRouter = new RoundRobinPartitionMessageRouterImpl(
                        conf.getHashingScheme(),
                        ThreadLocalRandom.current().nextInt(topicMetadata.numPartitions()),
                        conf.isBatchingEnabled(),
                        TimeUnit.MICROSECONDS.toMillis(conf.batchingPartitionSwitchFrequencyIntervalMicros()));
        }

        return messageRouter;
    }
}

具体实现比较简单就不介绍了。

看一下分区Producer调用发送代码：

public class PartitionedProducerImpl<T> {

    CompletableFuture<MessageId> internalSendWithTxnAsync(Message<?> message, Transaction txn) {
        // 路由实现类返回一个分区数
        int partition = routerPolicy.choosePartition(message, topicMetadata);
        
        // 获取对应分区Producer的实现 
        // 集合是按分区Producer创建顺序add的
        return producers.get(partition).internalSendWithTxnAsync(message, txn);
    }
}

上面调用internalSendWithTxnAsync是非分区ProducerImpl中的发送，后面都一样了，接下来不会再说分区和非分区了。

二、生产者发送实现

public class ProducerImpl<T> {

CompletableFuture<MessageId> internalSendAsync(Message<?> message) {
        CompletableFuture<MessageId> future = new CompletableFuture<>();
		// 创建生产者时是可以配置拦截器的，这里是发送前调用拦截器
        MessageImpl<?> interceptorMessage = (MessageImpl) beforeSend(message);
        // netty堆外内存的引用计数+1，意味着在没有收到发送成功响应时堆外内存空间一直占用
        interceptorMessage.getDataBuffer().retain();
        // 拦截返回的消息可能修改过，防止后面调用出现NPE，Properties如果空初始化Collections.emptyMap()
        if (interceptors != null) {
            interceptorMessage.getProperties();
        }
        // 发送消息，添加一个回调，当收到服务端响应成功时调用sendComplete
        sendAsync(interceptorMessage, new SendCallback() {
            SendCallback nextCallback = null;
            MessageImpl<?> nextMsg = null;
            long createdAt = System.nanoTime();

            @Override
            public CompletableFuture<MessageId> getFuture() {
                return future;
            }

            @Override
            public SendCallback getNextSendCallback() {
                return nextCallback;
            }

            @Override
            public MessageImpl<?> getNextMessage() {
                return nextMsg;
            }

            @Override
            public void sendComplete(Exception e) {
                try {
                    if (e != null) {
                        stats.incrementSendFailed();
                        // 调用拦截器的发送完成回调 成功失败都调用
                        onSendAcknowledgement(interceptorMessage, null, e);
                        future.completeExceptionally(e);
                    } else {
                    	// 调用拦截器的发送完成回调
                        onSendAcknowledgement(interceptorMessage, interceptorMessage.getMessageId(), null);
                        // 发送完成
                        future.complete(interceptorMessage.getMessageId());
						// 监控 统计时间
                        stats.incrementNumAcksReceived(System.nanoTime() - createdAt);
                    }
                } finally {
                // 释放堆外内存
                    interceptorMessage.getDataBuffer().release();
                }
				// 有多个发送回调 处理的意思跟上面一样
                while (nextCallback != null) {
                    SendCallback sendCallback = nextCallback;
                    MessageImpl<?> msg = nextMsg;
                    try {
                        msg.getDataBuffer().retain();
                        if (e != null) {
                            stats.incrementSendFailed();
                            onSendAcknowledgement(msg, null, e);
                            sendCallback.getFuture().completeExceptionally(e);
                        } else {
                            onSendAcknowledgement(msg, msg.getMessageId(), null);
                            sendCallback.getFuture().complete(msg.getMessageId());
                            stats.incrementNumAcksReceived(System.nanoTime() - createdAt);
                        }
                        nextMsg = nextCallback.getNextMessage();
                        nextCallback = nextCallback.getNextSendCallback();
                    } finally {
                        msg.getDataBuffer().release();
                    }
                }
            }

            @Override
            public void addCallback(MessageImpl<?> msg, SendCallback scb) {
                nextMsg = msg;
                nextCallback = scb;
            }
        });
        return future;
    }
}

可以看到重点都在sendAsync中

删减非正向核心代码例如异常处理，日志等

public class ProducerImpl<T> {
    public void sendAsync(Message<?> message, SendCallback callback) {

        MessageImpl<?> msg = (MessageImpl<?>) message;
        // 命名不规范，明明是获取MessageMetadata
        MessageMetadata msgMetadata = msg.getMessageBuilder();
        // 数据存在堆外内存
        ByteBuf payload = msg.getDataBuffer();
        // 数据总长度
        int uncompressedSize = payload.readableBytes();
		// 配置中会配最大等待数和满了是否阻塞
		// 这里就是校验是否达到最大等待数和达到最大后阻塞还是抛异常
        if (!canEnqueueRequest(callback, message.getSequenceId(), uncompressedSize)) {
            return;
        }
		
        ByteBuf compressedPayload = payload;
        boolean compressed = false;
		// 如果没有开启批量发送或者配置了延迟发送则压缩
        if (!isBatchMessagingEnabled() || msgMetadata.hasDeliverAtTime()) {
        	// 通过配置的压缩算法进行压缩
        	// 然后释放payload堆外内存，压缩后返回的是新的堆外内存
            compressedPayload = applyCompression(payload);
            // 压缩过
            compressed = true;
			// 如果没有开启分块并且压缩后的长度还超过最大的消息长度限制，则抛异常
			// 最大5M
            int compressedSize = compressedPayload.readableBytes();
            if (compressedSize > ClientCnx.getMaxMessageSize() && !this.conf.isChunkingEnabled()) {
            	...
                compressedPayload.release();
                return;
            }
        }
		// 如果消息是复制来的，抛异常。因为是复制的所以肯定持久化过了
        if (!msg.isReplicated() && msgMetadata.hasProducerName()) {
            compressedPayload.release();
            return;
        }
		// Schema校验
        if (!populateMessageSchema(msg, callback)) {
            compressedPayload.release();
            return;
        }
		// 如果开启了批量并且不是延迟发送则是1
		// 没有校验用户是否开启分块就开始分了，分块配置默认还是false
        int totalChunks = canAddToBatch(msg) ? 1
                : Math.max(1, compressedPayload.readableBytes()) / ClientCnx.getMaxMessageSize()
                        + (Math.max(1, compressedPayload.readableBytes()) % ClientCnx.getMaxMessageSize() == 0 ? 0 : 1);

        for (int i = 0; i < (totalChunks - 1); i++) {
        	// 这里目前是只会增加发送等待计数
            if (!canEnqueueRequest(callback, message.getSequenceId(), 0 /* The memory was already reserved */)) {
                return;
            }
        }

        try {
            synchronized (this) {
                int readStartIndex = 0;
                long sequenceId;
                // 获取SequenceId
                if (!msgMetadata.hasSequenceId()) {
                    sequenceId = msgIdGeneratorUpdater.getAndIncrement(this);
                    msgMetadata.setSequenceId(sequenceId);
                } else {
                    sequenceId = msgMetadata.getSequenceId();
                }
                String uuid = totalChunks > 1 ? String.format("%s-%d", producerName, sequenceId) : null;
                // 如果是分块，SequenceId都一样
                for (int chunkId = 0; chunkId < totalChunks; chunkId++) {
                	// readStartIndex就是分块的index，从哪里开始读，第一次是0，也就是0～5M
                    serializeAndSendMessage(msg, payload, sequenceId, uuid, chunkId, totalChunks,
                            readStartIndex, ClientCnx.getMaxMessageSize(), compressedPayload, compressed,
                            compressedPayload.readableBytes(), uncompressedSize, callback);
                            // 下一次是5M那个位置，也就是5M位置～10M
                    readStartIndex = ((chunkId + 1) * ClientCnx.getMaxMessageSize());
                }
            }
        } catch (PulsarClientException e) {
        } catch (Throwable t) {
        }
    }
}

可看到发送核心是最后的serializeAndSendMessage

public class ProducerImpl<T> {

// 要记得如果分块外面套了个循环的
private void serializeAndSendMessage(MessageImpl<?> msg, ByteBuf payload,
            long sequenceId, String uuid, int chunkId, int totalChunks, int readStartIndex, int chunkMaxSizeInBytes, ByteBuf compressedPayload,
            boolean compressed, int compressedPayloadSize,
            int uncompressedSize, SendCallback callback) throws IOException, InterruptedException {
        ByteBuf chunkPayload = compressedPayload;
        MessageMetadata msgMetadata = msg.getMessageBuilder();
        // 分块且持久化topic
        if (totalChunks > 1 && TopicName.get(topic).isPersistent()) {
        	// 从分配好的堆外内存中切出一块，从计算的index开始到5M，不足5M说明最后一个，与5对比取最小
            chunkPayload = compressedPayload.slice(readStartIndex,
                    Math.min(chunkMaxSizeInBytes, chunkPayload.readableBytes() - readStartIndex));
			// chunkId是第几块
			// 如果不是最后一块，引用计数+1
            if (chunkId != totalChunks - 1) {
                chunkPayload.retain();
            }
            if (uuid != null) {
                msgMetadata.setUuid(uuid);
            }
            // 消息元数据，设置当前第几块、总共分了几块、数据总共多大
            msgMetadata.setChunkId(chunkId)
                .setNumChunksFromMsg(totalChunks)
                .setTotalChunkMsgSize(compressedPayloadSize);
        }
        // 没有发送时间，说明第一次发
        if (!msgMetadata.hasPublishTime()) {
            msgMetadata.setPublishTime(client.getClientClock().millis());

            checkArgument(!msgMetadata.hasProducerName());

            msgMetadata.setProducerName(producerName);
			// 消息设置压缩方式，服务端将来解压
            if (conf.getCompressionType() != CompressionType.NONE) {
                msgMetadata
                        .setCompression(CompressionCodecProvider.convertToWireProtocol(conf.getCompressionType()));
            }
            // 没压缩前的大小
            msgMetadata.setUncompressedSize(uncompressedSize);
        }
		// 批量发送且没分块
        if (canAddToBatch(msg) && totalChunks <= 1) {
        	// 检查批量容器剩余空间是否足够
            if (canAddToCurrentBatch(msg)) {
				// 当前sequenceId比上一个小可能重复，可以开始去重
                if (sequenceId <= lastSequenceIdPushed) {
                    isLastSequenceIdPotentialDuplicated = true;
                    if (sequenceId <= lastSequenceIdPublished) {
                    } else {
                    }
                    // 添加并批量发送
                    // 批量发送容器有两个实现，一个key分组的，一个普通的
                    // 将一组消息进行压缩构建一个发送命令
                    doBatchSendAndAdd(msg, callback, payload);
                } else {
                	// 重复的和不重复的放到两批里
                    if (isLastSequenceIdPotentialDuplicated) {
                        doBatchSendAndAdd(msg, callback, payload);
                    } else {
                    	// 添加完如果满了直接发出去
                        boolean isBatchFull = batchMessageContainer.add(msg, callback);
                        lastSendFuture = callback.getFuture();
                        payload.release();
                        if (isBatchFull) {
                            batchMessageAndSend();
                        }
                    }
                    isLastSequenceIdPotentialDuplicated = false;
                }
            } else {
            	// 不足 先发送再添加
                doBatchSendAndAdd(msg, callback, payload);
            }
        } else {
        	// 如果前面没有执行压缩，这里执行
            if (!compressed) {
                chunkPayload = applyCompression(chunkPayload);
            }
            // 加密
            ByteBuf encryptedPayload = encryptMessage(msgMetadata, chunkPayload);
			// 批量大小，这里没有批量应该是1
            int numMessages = msg.getMessageBuilder().hasNumMessagesInBatch()
                    ? msg.getMessageBuilder().getNumMessagesInBatch()
                    : 1;
            final OpSendMsg op;
            // 构建发送请求，请求命令是cmd
            if (msg.getSchemaState() == MessageImpl.SchemaState.Ready) {
                ByteBufPair cmd = sendMessage(producerId, sequenceId, numMessages, msgMetadata, encryptedPayload);
                op = OpSendMsg.create(msg, cmd, sequenceId, callback);
            } else {
                op = OpSendMsg.create(msg, null, sequenceId, callback);
                final MessageMetadata finalMsgMetadata = msgMetadata;
                op.rePopulate = () -> {
                    op.cmd = sendMessage(producerId, sequenceId, numMessages, finalMsgMetadata, encryptedPayload);
                };
            }
            // 1
            op.setNumMessagesInBatch(numMessages);
            // 加密后的大小
            op.setBatchSizeByte(encryptedPayload.readableBytes());
            if (totalChunks > 1) {
                op.totalChunks = totalChunks;
                op.chunkId = chunkId;
            }
            lastSendFuture = callback.getFuture();
            // 处理发送
            processOpSendMsg(op);
        }
    }
}

上面关键的在最后一行processOpSendMsg(op);

public class ProducerImpl<T> {

protected void processOpSendMsg(OpSendMsg op) {
        if (op == null) {
            return;
        }
        try {
            if (op.msg != null && isBatchMessagingEnabled()) {
            	// 调用一次批量发送 内部递归直到批量容器为空
                batchMessageAndSend();
            }
            // 发送前添加到等待队列，等到收到服务端添加成功响应从中移除
            pendingMessages.add(op);
            if (op.msg != null) {
            	// 更新SequenceId
                LAST_SEQ_ID_PUSHED_UPDATER.getAndUpdate(this,
                        last -> Math.max(last, getHighestSequenceId(op)));
            }
            // 如果连接还是正常的
            if (shouldWriteOpSendMsg()) {
                ClientCnx cnx = cnx();
                // Schema没有
                if (op.msg != null && op.msg.getSchemaState() == None) {				
                	// 先注册Schema，内部等注册完再发送
                    tryRegisterSchema(cnx, op.msg, op.callback);
                    return;
                }
                // If we do have a connection, the message is sent immediately, otherwise we'll try again once a new
                // connection is established
                // 添加cmd引用计数
                op.cmd.retain();
                // 执行发送 
                cnx.ctx().channel().eventLoop().execute(WriteInEventLoopCallback.create(this, cnx, op));
                stats.updateNumMsgsSent(op.numMessagesInBatch, op.batchSizeByte);
            } else {
            	// 连接断了
            	// 没关系，等建立连接后open时会调用重发
            }
        } catch (Throwable t) {
            releaseSemaphoreForSendOp(op);
            op.sendComplete(new PulsarClientException(t, op.sequenceId));
        }
    }
}

以上就是客户端发送的实现，带大家回顾一下上篇介绍producer的创建时
Pulsar源码解析-非分区Topic的Producer底层实现原理

生产者的创建是：先寻找topic所在broker，然后打开连接，在打开连接的方法中有两块和上面相关的内容：
1、创建batchTimerTask定时任务，定时调用batchMessageAndSend
2、调用消息重发遍历pendingMessages