skywalking源码--轻量级队列内核

黑夜无路人

已于 2022-11-08 08:46:23 修改

阅读量423

点赞数

分类专栏： skywalking skywalking-agent 文章标签：轻量级队列内核 skywalking apm skywalking探针

于 2022-08-20 18:58:17 首次发布

本文链接：https://blog.csdn.net/SO_zxn/article/details/126441829

版权

skywalking 同时被 2 个专栏收录

12 篇文章 0 订阅

订阅专栏

skywalking-agent

10 篇文章 0 订阅

订阅专栏

本源码来自于skywalking-agent 8.9.0版本
本节主要讲解skywalking-agent的轻量级队列内核，该实现在datacarrier模块主要用于进行数据发送OAP服务端的实现，主要采用缓存批量异步发送的方式进行发送。
注：本篇文章主要是作为自己看书后的总结，内容有可能会存在一些个人理解上的偏差，如果有网友找出问题欢迎提出，感谢！！！如果我理解上的错误误导了您，在此表示抱歉！！！

文章目录

轻量级队列内核是基于无锁环状队列的生产者-消费者内存消息队列，主要作用是在生产者与消费者之间创建的一个缓冲的异步内存队列，防止因skywalking收集数据时生产端生产数据速度远大于消费端发送数据到OAP的速度，导致数据积压、生产阻塞。

源码分析

buffer 包

QueueBuffer接口

public interface QueueBuffer<T> {
    /**
     * Save data into the queue;
     *
     * @param data to add.
     * @return true if saved
     */
    boolean save(T data);

    /**
     * Set different strategy when queue is full.
     */
    void setStrategy(BufferStrategy strategy);

    /**
     * Obtain the existing data from the queue
     */
    void obtain(List<T> consumeList);

    int getBufferSize();
}

QueueBuffer为buffer实现的接口，这里的save方法是核心，里面涉及到保存策略

Buffer

是skywalking内存队列的核心载体，数据都存储在buffer的实例对象内。

public class Buffer<T> implements QueueBuffer<T> {
    private final Object[] buffer;
    private BufferStrategy strategy;
    private AtomicRangeInteger index;

    Buffer(int bufferSize, BufferStrategy strategy) {
        buffer = new Object[bufferSize];
        this.strategy = strategy;
        index = new AtomicRangeInteger(0, bufferSize);
    }

    @Override
    public void setStrategy(BufferStrategy strategy) {
        this.strategy = strategy;
    }

    @Override
    public boolean save(T data) {
        int i = index.getAndIncrement();
        // **buffer 写入位置，暂未被消费，已经存在值,这时候根据使用的存储策略进行处理**
        if (buffer[i] != null) {
            switch (strategy) {
                case IF_POSSIBLE:
                    return false;
                default:
            }
        }
        buffer[i] = data;
        return true;
    }

    @Override
    public int getBufferSize() {
        return buffer.length;
    }

    @Override
    public void obtain(List<T> consumeList) {
        this.obtain(consumeList, 0, buffer.length);
    }

    void obtain(List<T> consumeList, int start, int end) {
        for (int i = start; i < end; i++) {
            if (buffer[i] != null) {
                consumeList.add((T) buffer[i]);
                buffer[i] = null;
            }
        }
    }

}

Channels

是管理buffer的载体。

public class Channels<T> {
    private final QueueBuffer<T>[] bufferChannels;
    private IDataPartitioner<T> dataPartitioner;
    private final BufferStrategy strategy;
    private final long size;

    public Channels(int channelSize, int bufferSize, IDataPartitioner<T> partitioner, BufferStrategy strategy) {
        this.dataPartitioner = partitioner;
        this.strategy = strategy;
        bufferChannels = new QueueBuffer[channelSize];
        for (int i = 0; i < channelSize; i++) {
            if (BufferStrategy.BLOCKING.equals(strategy)) {
                bufferChannels[i] = new ArrayBlockingQueueBuffer<>(bufferSize, strategy);
            } else {
                bufferChannels[i] = new Buffer<>(bufferSize, strategy);
            }
        }
        // noinspection PointlessArithmeticExpression
        size = 1L * channelSize * bufferSize; // it's not pointless, it prevents numeric overflow before assigning an integer to a long
    }

    public boolean save(T data) {
        int index = dataPartitioner.partition(bufferChannels.length, data);
        int retryCountDown = 1;
        // **当缓冲策略为 BufferStrategy.IF_POSSIBLE 时，根据 IDataPartitioner 定义的重试次数，进行多次保存数据直到成功**
        if (BufferStrategy.IF_POSSIBLE.equals(strategy)) {
            int maxRetryCount = dataPartitioner.maxRetryCount();
            if (maxRetryCount > 1) {
                retryCountDown = maxRetryCount;
            }
        }
        for (; retryCountDown > 0; retryCountDown--) {
            if (bufferChannels[index].save(data)) {
                return true;
            }
        }
        return false;
    }

    public void setPartitioner(IDataPartitioner<T> dataPartitioner) {
        this.dataPartitioner = dataPartitioner;
    }

    /**
     * override the strategy at runtime. Notice, this will override several channels one by one. So, when running
     * setStrategy, each channel may use different BufferStrategy
     */
    public void setStrategy(BufferStrategy strategy) {
        for (QueueBuffer<T> buffer : bufferChannels) {
            buffer.setStrategy(strategy);
        }
    }

    /**
     * get channelSize
     */
    public int getChannelSize() {
        return this.bufferChannels.length;
    }

    public long size() {
        return size;
    }

    public QueueBuffer<T> getBuffer(int index) {
        return this.bufferChannels[index];
    }
}

partition包

该包下存在 IDataPartitioner 接口以及它的两个实现类ProducerThreadPartitioner、SimpleRollingPartitioner

IDataPartitioner

public interface IDataPartitioner<T> {
    int partition(int total, T data);

    /**
     * @return an integer represents how many times should retry when {@link BufferStrategy#IF_POSSIBLE}.
     * <p>
     * Less or equal 1, means not support retry.
     */
    int maxRetryCount();
}

partition(total, data) 接口方法，获得数据被分配的分区位置。
maxRetryCount() 接口方法，获得最大重试次数。

SimpleRollingPartitioner

public class SimpleRollingPartitioner<T> implements IDataPartitioner<T> {
    // **用于记录下一个当前分配的分区号**
    @SuppressWarnings("NonAtomicVolatileUpdate")
    private volatile int i = 0;

    @Override
    public int partition(int total, T data) {
        return Math.abs(i++ % total);
    }

    @Override
    public int maxRetryCount() {
        return 3;
    }
}

SimpleRollingPartitioner ，基于顺序分配策略的数据分配者实现类。

ProducerThreadPartitioner

public class ProducerThreadPartitioner<T> implements IDataPartitioner<T> {
    public ProducerThreadPartitioner() {
    }

    @Override
    public int partition(int total, T data) {
        return (int) Thread.currentThread().getId() % total;
    }

    @Override
    public int maxRetryCount() {
        return 1;
    }
}

ProducerThreadPartitioner ，基于线程编号分配策略的数据分配者实现类。

consumer包

主要包含 ConsumeDriver 、ConsumerThread 、IConsumer 三个类。

ConsumerThread 使用 IConsumer ，消费数据
ConsumeDriver 是 ConsumerThread 的线程池封装

IConsumer接口

消费者接口。定义了如下方法：

init() 接口方法，初始化消费者。
consume(List) 接口方法，批量消费消息。
onError(List, Throwable) 接口方法，处理当消费发生异常。
onExit() 接口方法，处理当消费结束。此处的结束时，ConsumerThread 关闭。

ConsumerThread

继承 java.lang.Thread ，消费线程。

public class ConsumerThread<T> extends Thread {
    private volatile boolean running;
    private IConsumer<T> consumer;
    private List<DataSource> dataSources;
    private long consumeCycle;

    ConsumerThread(String threadName, IConsumer<T> consumer, long consumeCycle) {
        super(threadName);
        this.consumer = consumer;
        running = false;
        dataSources = new ArrayList<DataSource>(1);
        this.consumeCycle = consumeCycle;
    }

    /**
     * add whole buffer to consume
     */
    void addDataSource(QueueBuffer<T> sourceBuffer) {
        this.dataSources.add(new DataSource(sourceBuffer));
    }

    @Override
    public void run() {
        running = true;

        final List<T> consumeList = new ArrayList<T>(1500);
        // **不断消费，直到线程关闭( #shutdown() )**
        while (running) {
        // consume(consumeList):批量消费数据
            if (!consume(consumeList)) {
                try {
                // **当未消费到数据，说明 dataSources 为空，等待 consumeCycle ms ，避免 CPU 空跑。**
                    Thread.sleep(consumeCycle);
                } catch (InterruptedException e) {
                }
            }
        }

        // consumer thread is going to stop
        // consume the last time
        // **当线程关闭，调用 consume() 方法，消费完 dataSources 剩余的数据**
        consume(consumeList);

       // onExit() 方法，处理当消费结束
        consumer.onExit();
    }

    private boolean consume(List<T> consumeList) {
        for (DataSource dataSource : dataSources) {
            dataSource.obtain(consumeList);
        }

        // 从 dataSources 中，获取要消费的数据
        if (!consumeList.isEmpty()) {
            try {
                // 当有数据可消费时，调用 consume(List<T>) 方法。
                consumer.consume(consumeList);
            } catch (Throwable t) {
               // 当消费发生异常时，调用 onError(List<T>, Throwable) 方法
                consumer.onError(consumeList, t);
            } finally {
                consumeList.clear();
            }
            return true;
        }
        consumer.nothingToConsume();
        return false;
    }

    void shutdown() {
        running = false;
    }

    /**
     * DataSource is a refer to {@link Buffer}.
     */
    class DataSource {
        private QueueBuffer<T> sourceBuffer;

        DataSource(QueueBuffer<T> sourceBuffer) {
            this.sourceBuffer = sourceBuffer;
        }

        void obtain(List<T> consumeList) {
            sourceBuffer.obtain(consumeList);
        }
    }
}

running 属性，是否运行中。
consumer 属性，消费者对象。
dataSources 属性，消费消息的数据源( DataSource )数组。一个 ConsumerThread ，可以消费多个 Buffer ，并且单个 Buffer 消费的分区范围可配置，即一个 Buffer 可以被多个 ConsumerThread 同时无冲突的消费。
addDataSource(sourceBuffer, start, end) 方法，添加 Buffer 部分范围。
addDataSource(sourceBuffer) 方法，添加 Buffer 全部范围。

ConsumeDriver

消费者驱动，提供了对 Channels 启动指定数量的 ConsumerThread 进行消费。

public class ConsumeDriver<T> implements IDriver {
    private boolean running;
    private ConsumerThread[] consumerThreads;
    private Channels<T> channels;
    private ReentrantLock lock;

    public ConsumeDriver(String name,
                         Channels<T> channels, Class<? extends IConsumer<T>> consumerClass,
                         int num,
                         long consumeCycle,
                         Properties properties) {
        this(channels, num);
        for (int i = 0; i < num; i++) {
            consumerThreads[i] = new ConsumerThread(
                "DataCarrier." + name + ".Consumer." + i + ".Thread", getNewConsumerInstance(consumerClass, properties),
                consumeCycle
            );
            consumerThreads[i].setDaemon(true);
        }
    }

    public ConsumeDriver(String name, Channels<T> channels, IConsumer<T> prototype, int num, long consumeCycle) {
        this(channels, num);
        prototype.init(new Properties());
        for (int i = 0; i < num; i++) {
            consumerThreads[i] = new ConsumerThread(
                "DataCarrier." + name + ".Consumer." + i + ".Thread", prototype, consumeCycle);
            consumerThreads[i].setDaemon(true);
        }

    }

    private ConsumeDriver(Channels<T> channels, int num) {
        running = false;
        this.channels = channels;
        consumerThreads = new ConsumerThread[num];
        lock = new ReentrantLock();
    }

    private IConsumer<T> getNewConsumerInstance(Class<? extends IConsumer<T>> consumerClass, Properties properties) {
        try {
            IConsumer<T> inst = consumerClass.getDeclaredConstructor().newInstance();
            inst.init(properties);
            return inst;
        } catch (InstantiationException e) {
            throw new ConsumerCannotBeCreatedException(e);
        } catch (IllegalAccessException e) {
            throw new ConsumerCannotBeCreatedException(e);
        } catch (NoSuchMethodException e) {
            throw new ConsumerCannotBeCreatedException(e);
        } catch (InvocationTargetException e) {
            throw new ConsumerCannotBeCreatedException(e);
        }
    }

    @Override
    public void begin(Channels channels) {
        // **正在运行中，直接返回**
        if (running) {
            return;
        }
        // 获得锁
        lock.lock();
        try {
            // **调用 #allocateBuffer2Thread() 方法，将 channels 的多个 Buffer ，分配给 consumerThreads 的多个 ConsumerThread。**
            this.allocateBuffer2Thread();
            for (ConsumerThread consumerThread : consumerThreads) {
                // **启动每个 ConsumerThread ，开始消费。**
                consumerThread.start();
            }
            // **标记正在运行中**
            running = true;
        } finally {
            // **释放锁**
            lock.unlock();
        }
    }

    @Override
    public boolean isRunning(Channels channels) {
        return running;
    }

    private void allocateBuffer2Thread() {
        int channelSize = this.channels.getChannelSize();
        /**
         * if consumerThreads.length < channelSize
         * each consumer will process several channels.
         *
         * if consumerThreads.length == channelSize
         * each consumer will process one channel.
         *
         * if consumerThreads.length > channelSize
         * there will be some threads do nothing.
         */
         // **轮巡将每个channel分配给consumerThread，这里注意如果consumerThread大于channel那么会出现大于channel的那部分channel会没有channel可以消费。如果channel大于consumerThread，那么编号位于前面的consumerThread将多消费channel。**
        for (int channelIndex = 0; channelIndex < channelSize; channelIndex++) {
            int consumerIndex = channelIndex % consumerThreads.length;
            consumerThreads[consumerIndex].addDataSource(channels.getBuffer(channelIndex));
        }

    }

    @Override
    public void close(Channels channels) {
        // **获得锁**
        lock.lock();
        try {
            // **标记不在运行中**
            this.running = false;
            for (ConsumerThread consumerThread : consumerThreads) {
                // **关闭每个 ConsumerThread ，结束消费**
                consumerThread.shutdown();
            }
        } finally {
            // **释放锁**
            lock.unlock();
        }
    }
}

running 属性，是否运行中。
consumerThreads 属性，ConsumerThread 数组，通过构造方法的 num 参数进行指定。
channels 属性，数据通道。
lock 属性，锁。保证 ConsumerPool 启动或关闭时的线程安全。

begin() 方法，启动 ConsumerPool ，进行数据消费。
close() 方法，关闭 ConsumerPool 。

DataCarrier

DataCarrier 异步处理库的入口程序。通过创建 DataCarrier 对象，使用生产者消费者的模式，执行异步执行逻辑。它就是将上面的 consumer、channel、partition整合在一起的数据搬运工类。初始化时需要设置channel的大小和每个channel内buffer的个数。

public class ConsumeDriver<T> implements IDriver {
    private boolean running;
    private ConsumerThread[] consumerThreads;
    private Channels<T> channels;
    private ReentrantLock lock;

    public ConsumeDriver(String name,
                         Channels<T> channels, Class<? extends IConsumer<T>> consumerClass,
                         int num,
                         long consumeCycle,
                         Properties properties) {
        this(channels, num);
        for (int i = 0; i < num; i++) {
            consumerThreads[i] = new ConsumerThread(
                "DataCarrier." + name + ".Consumer." + i + ".Thread", getNewConsumerInstance(consumerClass, properties),
                consumeCycle
            );
            consumerThreads[i].setDaemon(true);
        }
    }

    public ConsumeDriver(String name, Channels<T> channels, IConsumer<T> prototype, int num, long consumeCycle) {
        this(channels, num);
        prototype.init(new Properties());
        for (int i = 0; i < num; i++) {
            consumerThreads[i] = new ConsumerThread(
                "DataCarrier." + name + ".Consumer." + i + ".Thread", prototype, consumeCycle);
            consumerThreads[i].setDaemon(true);
        }

    }

    private ConsumeDriver(Channels<T> channels, int num) {
        running = false;
        this.channels = channels;
        consumerThreads = new ConsumerThread[num];
        lock = new ReentrantLock();
    }

    private IConsumer<T> getNewConsumerInstance(Class<? extends IConsumer<T>> consumerClass, Properties properties) {
        try {
            IConsumer<T> inst = consumerClass.getDeclaredConstructor().newInstance();
            inst.init(properties);
            return inst;
        } catch (InstantiationException e) {
            throw new ConsumerCannotBeCreatedException(e);
        } catch (IllegalAccessException e) {
            throw new ConsumerCannotBeCreatedException(e);
        } catch (NoSuchMethodException e) {
            throw new ConsumerCannotBeCreatedException(e);
        } catch (InvocationTargetException e) {
            throw new ConsumerCannotBeCreatedException(e);
        }
    }

    @Override
    public void begin(Channels channels) {
        if (running) {
            return;
        }
        lock.lock();
        try {
            this.allocateBuffer2Thread();
            for (ConsumerThread consumerThread : consumerThreads) {
                consumerThread.start();
            }
            running = true;
        } finally {
            lock.unlock();
        }
    }

    @Override
    public boolean isRunning(Channels channels) {
        return running;
    }

    private void allocateBuffer2Thread() {
        int channelSize = this.channels.getChannelSize();
        /**
         * if consumerThreads.length < channelSize
         * each consumer will process several channels.
         *
         * if consumerThreads.length == channelSize
         * each consumer will process one channel.
         *
         * if consumerThreads.length > channelSize
         * there will be some threads do nothing.
         */
        for (int channelIndex = 0; channelIndex < channelSize; channelIndex++) {
            int consumerIndex = channelIndex % consumerThreads.length;
            consumerThreads[consumerIndex].addDataSource(channels.getBuffer(channelIndex));
        }

    }

    @Override
    public void close(Channels channels) {
        lock.lock();
        try {
            this.running = false;
            for (ConsumerThread consumerThread : consumerThreads) {
                consumerThread.shutdown();
            }
        } finally {
            lock.unlock();
        }
    }
}