2.RecordAccumulator之BufferPool设计

RecordAccumulator之BufferPool设计

上节我们讲解了kafka基本概念已经部分原理,之后我们将要从Producer->Brokers->Consumer分别进行分析,Producer首先要讲解的就是RecordAccumulator是如何实现队列,并对消息进行缓存的.本节我们先了解BufferPool是如何分配资源的.

BufferPool中的变量
//总内存由buffer.memory配置
private final long totalMemory;
//队列中的ByteBuffer大小,由batch.size配置
private final int poolableSize;
private final ReentrantLock lock;
//已申请空间,但未使用
private final Deque<ByteBuffer> free;
//申请不到足够内存而阻塞的线程
private final Deque<Condition> waiters;
/** Total available memory is the sum of nonPooledAvailableMemory and the number of byte buffers in free * poolableSize.  */
//可用内存=nonPooledAvailableMemory+free*buffers.size
private long nonPooledAvailableMemory;

其中BufferPool分配空间的逻辑如下:

申请size大小的方式:1.申请超过totalMemory,异常结束.2.直接使用free队列中的ByteBuffer 3.用availableMemory方式分配(nonPooledAvailableMemory)

蓝色框内的为大多数的内存分配,就是从free队列中拿到想要的ByteBuffer,这是kafka最希望的内存分配方式(减少分配的额外消耗)

黄色的框为分配内存时队列中的内存不符合其分配的条件(队列为空或者消息大小不匹配等)这块的内存需要从从availableMemory中分配

绿色的框为当前内存不足,需要等待释放资源的情况,这里的实现逻辑就是accumulated初始化为0,当accumulated<size时进行阻塞,当内存释放时,唤醒阻塞的线程,将可以分配的内存使用accumulated进行累加,如果没有达到size大小,则继续阻塞,在等待内存释放.这里有一个逻辑如果accumulated=0并且消息大小符合要求,队列不为空条件会优先使用free队列

public ByteBuffer allocate(int size, long maxTimeToBlockMs) throws InterruptedException {
        if (size > this.totalMemory)
            throw new IllegalArgumentException("Attempt to allocate " + size
                                               + " bytes, but there is a hard limit of "
                                               + this.totalMemory
                                               + " on memory allocations.");

        ByteBuffer buffer = null;
        this.lock.lock();

        if (this.closed) {
            this.lock.unlock();
            throw new KafkaException("Producer closed while allocating memory");
        }

        try {
            // check if we have a free buffer of the right size pooled
            // 检验消息是否符合一条消息的大小(只有相同大小的消息才能存储在free队列中)
            if (size == poolableSize && !this.free.isEmpty())
                return this.free.pollFirst();

            // now check if the request is immediately satisfiable with the
            // memory on hand or if we need to block
            int freeListSize = freeSize() * this.poolableSize;
            // 可用内存 >消息的大小,这里可以直接分配内存.
            if (this.nonPooledAvailableMemory + freeListSize >= size) {
                // we have enough unallocated or pooled memory to immediately
                // satisfy the request, but need to allocate the buffer
                // 释放队列中的内存到nonPooledAvailableMemory中
                freeUp(size);
                // 减少size大小的内存(因为已经分配给size了)
                this.nonPooledAvailableMemory -= size;
            } else {
                // we are out of memory and will have to block
                // 累加器,大小从0开始
                int accumulated = 0;
                Condition moreMemory = this.lock.newCondition();
                try {
                    long remainingTimeToBlockNs = TimeUnit.MILLISECONDS.toNanos(maxTimeToBlockMs);
                    this.waiters.addLast(moreMemory);
                    // loop over and over until we have a buffer or have reserved
                    // enough memory to allocate one
                    // 累加器的大小一直小于size,则循环等待资源释放,等accumulate>=size时
                    while (accumulated < size) {
                        long startWaitNs = time.nanoseconds();
                        long timeNs;
                        boolean waitingTimeElapsed;
                        try {
                            waitingTimeElapsed = !moreMemory.await(remainingTimeToBlockNs, TimeUnit.NANOSECONDS);
                        } finally {
                            long endWaitNs = time.nanoseconds();
                            timeNs = Math.max(0L, endWaitNs - startWaitNs);
                            recordWaitTime(timeNs);
                        }

                        if (this.closed)
                            throw new KafkaException("Producer closed while allocating memory");

                        if (waitingTimeElapsed) {
                            this.metrics.sensor("buffer-exhausted-records").record();
                            throw new BufferExhaustedException("Failed to allocate memory within the configured max blocking time " + maxTimeToBlockMs + " ms.");
                        }

                        remainingTimeToBlockNs -= timeNs;

                        // check if we can satisfy this request from the free list,
                        // otherwise allocate memory
                        // 检查是否满足free里分配内存的请求,要是不满足就要分配内存了
                        if (accumulated == 0 && size == this.poolableSize && !this.free.isEmpty()) {
                            // just grab a buffer from the free list
                            // 满足则取第一个
                            buffer = this.free.pollFirst();
                            accumulated = size;
                        } else {
                            // we'll need to allocate memory, but we may only get
                            // part of what we need on this iteration
                            // 以上资源没有释放够,看free队列中是否可以释放多余的内存,我们需要分配内存,但是可能只需要申请一部分
                            freeUp(size - accumulated);
                            // 累加器已经等待释放的内存大小与nonPooled可用内存的比较,获取最小值
                            int got = (int) Math.min(size - accumulated, this.nonPooledAvailableMemory);
                            // 将nonPooled 中的内存分配给get
                            this.nonPooledAvailableMemory -= got;
                            accumulated += got;
                        }
                    }
                    // Don't reclaim memory on throwable since nothing was thrown
                    accumulated = 0;
                } finally {
                    // When this loop was not able to successfully terminate don't loose available memory
                    this.nonPooledAvailableMemory += accumulated;
                    this.waiters.remove(moreMemory);
                }
            }
        } finally {
            // signal any additional waiters if there is more memory left
            // over for them
            try {
                if (!(this.nonPooledAvailableMemory == 0 && this.free.isEmpty()) && !this.waiters.isEmpty())
                    this.waiters.peekFirst().signal();
            } finally {
                // Another finally... otherwise find bugs complains
                lock.unlock();
            }
        }

        if (buffer == null)
            return safeAllocateByteBuffer(size);
        else
            return buffer;
    }

总结:以上我们知道了BufferPool的设计只能针对于特定大小(poolableSize)的ByteBuffer进行管理,如果大小不相等则不会放入free队列中,如果大于poolableSize消息比较多,就不能更好的利用kafka实现的free内存池,频繁的内存回收则会降低使用率,并可能带来频繁的GC,我们就需要调整好batch.size 参数大小,酌情调优.