Disruptor学习笔记：基本使用、核心概念和原理

最新推荐文章于 2023-10-19 13:08:46 发布

邋遢的流浪剑客

最新推荐文章于 2023-10-19 13:08:46 发布

阅读量1.5k

点赞数 2

文章标签： Disruptor

本文链接：https://blog.csdn.net/qq_40378034/article/details/120387964

版权

Java多线程专栏收录该内容

34 篇文章 10 订阅

订阅专栏

Disruptor是英国外汇交易公司LMAX开发的一个高性能队列（系统内部的内存队列），研发的初衷是解决内存队列的延迟问题（在性能测试中发现竟然与I/O操作处于同样的数量级）。基于Disruptor开发的系统单线程能支撑每秒600万订单，2010年在QCon演讲后，获得了业界关注。2011年，企业应用软件专家Martin Fowler专门撰写长文介绍。同年它还获得了Oracle官方的Duke大奖

1、快速入门

1）、步骤

创建一个工厂Event类，用于创建Event类实例对象
需要有一个监听事件类，用于处理数据（Event类）
实例化Disruptor实例，配置一系列参数，编写Disruptor核心组件
编写生产者组件，向Disruptor容器中去投递数据

2）、引入依赖

        <dependency>
            <groupId>com.lmax</groupId>
            <artifactId>disruptor</artifactId>
            <version>3.3.2</version>
        </dependency>

3）、代码实现

1）基础元素工厂类

@Data
public class OrderEvent {
    private Long price;
}

public class OrderEventFactory implements EventFactory<OrderEvent> {
    @Override
    public OrderEvent newInstance() {
        // 返回空的数据对象(Event)
        return new OrderEvent();
    }
}

2）消费端事件处理器

public class OrderEventHandler implements EventHandler<OrderEvent> {
    @Override
    public void onEvent(OrderEvent event, long sequence, boolean endOfBatch) throws Exception {
        System.out.println("消费者:" + event.getPrice());
    }
}

3）构建Disruptor实例

public class Main {
    public static void main(String[] args) {
        // 1.实例化disruptor对象
        ExecutorService executor = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
        /**
         * 1 eventFactory:消息(Event)工厂对象
         * 2 ringBufferSize:容器的长度
         * 3 executor:线程池
         * 4 producerType:生产消费类型 单生产者还是多生产者
         * 5 waitStrategy:等待策略
         */
        Disruptor<OrderEvent> disruptor = new Disruptor<>(
                new OrderEventFactory(),
                1024 * 1024,
                executor,
                ProducerType.SINGLE,
                new BlockingWaitStrategy()
        );
        // 2.添加消费者的监听(构建disruptor与消费者的一个关联关系)
        disruptor.handleEventsWith(new OrderEventHandler());
        // 3.启动disruptor
        disruptor.start();
        // 4.获取实际存储数据的容器:ringBuffer
        RingBuffer<OrderEvent> ringBuffer = disruptor.getRingBuffer();
        OrderEventProducer producer = new OrderEventProducer(ringBuffer);
        for (long i = 0; i < 100; ++i) {
            producer.sendData(i);
        }
        disruptor.shutdown();
        executor.shutdown();
    }
}

4）生产者组件投递数据

public class OrderEventProducer {
    private RingBuffer<OrderEvent> ringBuffer;

    public OrderEventProducer(RingBuffer<OrderEvent> ringBuffer) {
        this.ringBuffer = ringBuffer;
    }

    public void sendData(Long price) {
        // 1.在生产者发送消息的时候,首先需要从ringBuffer中获取一个可用的序号
        long sequence = ringBuffer.next();
        try {
            // 2.根据这个序号,找到具体的OrderEvent元素 注意:此时获取的OrderEvent对象是一个属性没有被赋值的对象
            OrderEvent orderEvent = ringBuffer.get(sequence);
            // 3.进行实际的赋值
            orderEvent.setPrice(price);
        } finally {
            // 4.提交发布操作
            ringBuffer.publish(sequence);
        }
    }
}

2、Disruptor核心概念

1）、生产消费模型

生产者向RingBuffer中写入元素，消费者从RingBuffer中消费元素

在这里插入图片描述

2）、RingBuffer到底是什么？

核心概念：

RingBuffer：基于数据的缓存实现，也是创建sequencer与定义WaitStrategy的入口
Disruptor：持有RingBuffer、消费者线程池Executor、消费者集合ConsumerRepository等引用

RingBuffer是一个环（首位相接的环），用于在不同上下文（线程）间传递数据的buffer

在这里插入图片描述

RingBuffer拥有一个序号，这个序号指向数组中下一个可用元素

在这里插入图片描述

随着你不停地填充这个buffer（可能也会有相应的读取），这个序号会一直增长，直到绕过这个环

在这里插入图片描述

要找到数组中当前序号指向的元素，可以通过mod操作：sequence mod array length = array index

以上面的RingBuffer为例：12 % 10 = 2

事实上，上图中的RingBuffer只有10个槽完全是个意外。如果槽的个数是2的N次方更有利于基于二进制的计算机进行计算

3）、Sequence、Sequencer、SequenceBarrier

Sequence：

通过顺序递增的序号来编号，管理进行交换的数据（事件）

对数据（事件）的处理过程总是沿着序号逐个递增处理

一个Sequence用于跟踪标识某个特定的事件处理者（RingBuffer/Producer/Consumer）的处理进度

Sequence可以看成一个AtomicLong，用于标识进度

防止不同Sequence之间CPU缓存伪共享的问题

Sequencer：

Sequencer是Disruptor的真正核心

Sequencer有两个实现类：

SingleProducerSequencer（单生产者）
MultiProducerSequencer（多生产者）

主要实现生产者和消费者之间快速、正确地传递数据的并发算法

Sequence Barrier：

用于保持对RingBuffer的Producer和Consumer之间的平衡关系；Sequence Barrier还定义了决定Consumer是否还有可处理的事件的逻辑

4）、WaitStrategy消费者等待策略

WaitStrategy决定消费者将如何等待生产者将Event置入Disruptor

主要策略有：

BlockingWaitStrategy：是最低效的策略，但其对CPU的消耗最小并且在各种不同部署环境中能提供更加一致的性能表现
SleepingWaitStrategy：性能表现跟BlockingWaitStrategy差不多，对CPU的消耗也类似，但其对生产者线程的影响最小，适合用于异步日志类似的场景
YieldingWaitStrategy：性能是最好的，适合用于低延迟的系统。在要求极高性能且事件处理数小于CPU逻辑核心数的场景中，推荐使用此策略；例如，CPU开启超线程的特性

5）、EventProcessor、WorkProcessor等

Event：

从生产者到消费者过程中所处理的数据单元

Disruptor中没有代码表示Event，因为它完全由用户定义的

EventProcessor：

主要事件循环，处理Disruptor中的Event，拥有消费者的Sequence

它有一个实现类是BatchEventProcessor，包含了event loop有效的实现，并且将回调一个EventHandler接口的实现对象

EventHandler：

由用户实现并且代表了Disruptor中的一个消费者的接口，也就是我们的消费者逻辑都需要写在这里

WorkProcessor：

确保每个Sequence只被一个Processor消费，在同一个WorkPool中处理多个WorkProcessor不会消费同样的Sequence

6）、核心概念整体图解

在这里插入图片描述

3、Disruptor高级特性

1）、串、并行操作

EventHandlerGroup<T> handleEventsWith(final EventHandler<? super T>... handlers)

串行操作：使用链式调用的方式

并行操作：使用单独调用的方式

1）串行操作

@Data
public class Trade {
    private String id;

    private String name;

    private double price;

    private AtomicInteger count = new AtomicInteger(0);
}

消费端事件处理器：

public class Handler1 implements EventHandler<Trade> {
    @Override
    public void onEvent(Trade event, long sequence, boolean endOfBatch) throws Exception {
        System.out.println("handler1:set id");
        event.setId(UUID.randomUUID().toString());
        TimeUnit.SECONDS.sleep(1);
    }
}

public class Handler2 implements EventHandler<Trade> {
    @Override
    public void onEvent(Trade event, long sequence, boolean endOfBatch) throws Exception {
        System.out.println("handler2:set name");
        event.setName("h2");
        TimeUnit.SECONDS.sleep(1);
    }
}

public class Handler3 implements EventHandler<Trade> {
    @Override
    public void onEvent(Trade event, long sequence, boolean endOfBatch) throws Exception {
        System.out.println("handler3: name=" + event.getName() +
                ",id=" + event.getId() +
                ",instance:" + event.toString());
        TimeUnit.SECONDS.sleep(1);
    }
}

生产者组件投递数据：

public class TradeProducer implements Runnable {
    private CountDownLatch latch;

    private Disruptor<Trade> disruptor;

    private final static int PUBLISH_COUNT = 1;

    public TradeProducer(CountDownLatch latch, Disruptor<Trade> disruptor) {
        this.latch = latch;
        this.disruptor = disruptor;
    }

    @Override
    public void run() {
        for (int i = 0; i < PUBLISH_COUNT; ++i) {
            disruptor.publishEvent(new TradeEventTranslator());
        }
        latch.countDown();
    }
}

class TradeEventTranslator implements EventTranslator<Trade> {
    private Random random = new Random();

    @Override
    public void translateTo(Trade event, long sequence) {
        this.generateTrade(event);
    }

    private void generateTrade(Trade event) {
        event.setPrice(random.nextDouble() * 9999);
    }
}

串行操作：

public class Main {
    public static void main(String[] args) throws InterruptedException {
        ExecutorService es1 = Executors.newFixedThreadPool(4);
        ExecutorService es2 = Executors.newFixedThreadPool(4);
        // 1.构建Disruptor
        Disruptor<Trade> disruptor = new Disruptor<>(
                new EventFactory<Trade>() {
                    @Override
                    public Trade newInstance() {
                        return new Trade();
                    }
                },
                1024 * 1024,
                es1,
                ProducerType.SINGLE,
                new BusySpinWaitStrategy()
        );

        // 2.把消费者设置到Disruptor中
        // 2.1串行操作
        disruptor
                .handleEventsWith(new Handler1())
                .handleEventsWith(new Handler2())
                .handleEventsWith(new Handler3());

        // 3.启动disruptor
        disruptor.start();

        CountDownLatch latch = new CountDownLatch(1);
        long begin = System.currentTimeMillis();
        es2.submit(new TradeProducer(latch, disruptor));
        latch.await();

        disruptor.shutdown();
        es1.shutdown();
        es2.shutdown();
        System.out.println("总耗时:" + (System.currentTimeMillis() - begin));
    }
}

串行操作，先由Handler1处理，再交给Handler2处理，再交给Handler3处理，运行结果如下：

handler1:set id
handler2:set name
handler3: name=h2,id=94232ca3-bb07-445a-9969-1bae5cf2ad1b,instance:Trade(id=94232ca3-bb07-445a-9969-1bae5cf2ad1b, name=h2, price=6046.9618214089205, count=0)
总耗时:3100

2）并行操作

public class Main {
    public static void main(String[] args) throws InterruptedException {
        ExecutorService es1 = Executors.newFixedThreadPool(4);
        ExecutorService es2 = Executors.newFixedThreadPool(4);
        // 1.构建Disruptor
        Disruptor<Trade> disruptor = new Disruptor<>(
                new EventFactory<Trade>() {
                    @Override
                    public Trade newInstance() {
                        return new Trade();
                    }
                },
                1024 * 1024,
                es1,
                ProducerType.SINGLE,
                new BusySpinWaitStrategy()
        );

        // 2.把消费者设置到Disruptor中
        // 2.1串行操作
//        disruptor
//                .handleEventsWith(new Handler1())
//                .handleEventsWith(new Handler2())
//                .handleEventsWith(new Handler3());
        // 2.2并行操作
        disruptor.handleEventsWith(new Handler1(), new Handler2(), new Handler3());

        // 3.启动disruptor
        disruptor.start();

        CountDownLatch latch = new CountDownLatch(1);
        long begin = System.currentTimeMillis();
        es2.submit(new TradeProducer(latch, disruptor));
        latch.await();

        disruptor.shutdown();
        es1.shutdown();
        es2.shutdown();
        System.out.println("总耗时:" + (System.currentTimeMillis() - begin));
    }
}

并行操作，运行结果如下：

handler1:set id
handler2:set name
handler3: name=null,id=null,instance:Trade(id=null, name=null, price=7631.491448398892, count=0)
总耗时:1085

2）、菱形操作

Disruptor可以实现串并行同时编码：

在这里插入图片描述

Handler1和Handler2先并行执行，再串行执行Handler3

public class Main {
    public static void main(String[] args) throws InterruptedException {
        ExecutorService es1 = Executors.newFixedThreadPool(4);
        ExecutorService es2 = Executors.newFixedThreadPool(4);
        // 1.构建Disruptor
        Disruptor<Trade> disruptor = new Disruptor<>(
                new EventFactory<Trade>() {
                    @Override
                    public Trade newInstance() {
                        return new Trade();
                    }
                },
                1024 * 1024,
                es1,
                ProducerType.SINGLE,
                new BusySpinWaitStrategy()
        );

        // 2.把消费者设置到Disruptor中
        // 2.1串行操作
//        disruptor
//                .handleEventsWith(new Handler1())
//                .handleEventsWith(new Handler2())
//                .handleEventsWith(new Handler3());
        // 2.2并行操作
//        disruptor.handleEventsWith(new Handler1(), new Handler2(), new Handler3());
        // 2.3菱形操作1
//        disruptor
//                .handleEventsWith(new Handler1(), new Handler2())
//                .handleEventsWith(new Handler3());
        // 2.3菱形操作2
        EventHandlerGroup<Trade> ehGroup = disruptor.handleEventsWith(new Handler1(), new Handler2());
        ehGroup.then(new Handler3());

        // 3.启动disruptor
        disruptor.start();

        CountDownLatch latch = new CountDownLatch(1);
        long begin = System.currentTimeMillis();
        es2.submit(new TradeProducer(latch, disruptor));
        latch.await();

        disruptor.shutdown();
        es1.shutdown();
        es2.shutdown();
        System.out.println("总耗时:" + (System.currentTimeMillis() - begin));
    }
}

菱形操作，运行结果如下：

handler2:set name
handler1:set id
handler3: name=h2,id=54820d86-4bc5-4da4-a4a2-1f9e1d673006,instance:Trade(id=54820d86-4bc5-4da4-a4a2-1f9e1d673006, name=h2, price=4929.999989670335, count=0)
总耗时:2093

3）、多边形操作

在这里插入图片描述

消费端事件处理器：

public class Handler4 implements EventHandler<Trade> {
    @Override
    public void onEvent(Trade event, long sequence, boolean endOfBatch) throws Exception {
        System.out.println("handler4: name=" + event.getName() +
                ",id=" + event.getId() +
                ",instance:" + event.toString());
        TimeUnit.SECONDS.sleep(1);
    }
}

public class Handler5 implements EventHandler<Trade> {
    @Override
    public void onEvent(Trade event, long sequence, boolean endOfBatch) throws Exception {
        System.out.println("handler5: name=" + event.getName() +
                ",id=" + event.getId() +
                ",instance:" + event.toString());
        TimeUnit.SECONDS.sleep(1);
    }
}

多边形操作：

public class Main {
    public static void main(String[] args) throws InterruptedException {
        ExecutorService es1 = Executors.newFixedThreadPool(5);
        ExecutorService es2 = Executors.newFixedThreadPool(1);
        // 1.构建Disruptor
        Disruptor<Trade> disruptor = new Disruptor<>(
                new EventFactory<Trade>() {
                    @Override
                    public Trade newInstance() {
                        return new Trade();
                    }
                },
                1024 * 1024,
                es1,
                ProducerType.SINGLE,
                new BusySpinWaitStrategy()
        );

        // 2.把消费者设置到Disruptor中
        // 2.1串行操作
//        disruptor
//                .handleEventsWith(new Handler1())
//                .handleEventsWith(new Handler2())
//                .handleEventsWith(new Handler3());
        // 2.2并行操作
//        disruptor.handleEventsWith(new Handler1(), new Handler2(), new Handler3());
        // 2.3菱形操作1
//        disruptor
//                .handleEventsWith(new Handler1(), new Handler2())
//                .handleEventsWith(new Handler3());
        // 2.3菱形操作2
//        EventHandlerGroup<Trade> ehGroup = disruptor.handleEventsWith(new Handler1(), new Handler2());
//        ehGroup.then(new Handler3());
        // 2.4 六边形操作
        // 有5个事件监听 调整es1线程数为5
        Handler1 h1 = new Handler1();
        Handler2 h2 = new Handler2();
        Handler3 h3 = new Handler3();
        Handler4 h4 = new Handler4();
        Handler5 h5 = new Handler5();
        disruptor.handleEventsWith(h1, h4);
        disruptor.after(h1).handleEventsWith(h2);
        disruptor.after(h4).handleEventsWith(h5);
        disruptor.after(h2, h5).handleEventsWith(h3);

        // 3.启动disruptor
        disruptor.start();

        CountDownLatch latch = new CountDownLatch(1);
        long begin = System.currentTimeMillis();
        es2.submit(new TradeProducer(latch, disruptor));
        latch.await();

        disruptor.shutdown();
        es1.shutdown();
        es2.shutdown();
        System.out.println("总耗时:" + (System.currentTimeMillis() - begin));
    }
}

多边形操作，运行结果如下：

handler1:set id
handler4: name=null,id=null,instance:Trade(id=null, name=null, price=2773.0238888862746, count=0)
handler5: name=null,id=97ec7a78-119b-4309-a07b-b103d466bbc7,instance:Trade(id=97ec7a78-119b-4309-a07b-b103d466bbc7, name=null, price=2773.0238888862746, count=0)
handler2:set name
handler3: name=h2,id=97ec7a78-119b-4309-a07b-b103d466bbc7,instance:Trade(id=97ec7a78-119b-4309-a07b-b103d466bbc7, name=h2, price=2773.0238888862746, count=0)
总耗时:3057

为什么5个Handler需要5个线程？

public class Disruptor<T>
{
    private final Executor executor;

    public RingBuffer<T> start()
    {
        final Sequence[] gatingSequences = consumerRepository.getLastSequenceInChain(true);
        ringBuffer.addGatingSequences(gatingSequences);

        checkOnlyStartedOnce();
      	// 循环 有多少个消费者调用多少次
        for (final ConsumerInfo consumerInfo : consumerRepository)
        {
            consumerInfo.start(executor);
        }

        return ringBuffer;
    }

class EventProcessorInfo<T> implements ConsumerInfo
{
    private final EventProcessor eventprocessor;
  
    @Override
    public void start(final Executor executor)
    {
      	// 执行EventProcessor
        executor.execute(eventprocessor);
    }

EventProcessor实现了Runnable接口，批量操作由EventProcessor的子类BatchEventProcessor（单消费者使用）来处理

public final class BatchEventProcessor<T>
    implements EventProcessor
{
  
    @Override
    public void run()
    {
        if (!running.compareAndSet(false, true))
        {
            throw new IllegalStateException("Thread is already running");
        }
        sequenceBarrier.clearAlert();

        notifyStart();

        T event = null;
        long nextSequence = sequence.get() + 1L;
        try
        {
            while (true)
            {
                try
                {
                    final long availableSequence = sequenceBarrier.waitFor(nextSequence);

                    while (nextSequence <= availableSequence)
                    {
                        event = dataProvider.get(nextSequence);
                      	// 调用具体消费端的逻辑
                        eventHandler.onEvent(event, nextSequence, nextSequence == availableSequence);
                        nextSequence++;
                    }

                    sequence.set(availableSequence);
                }
                catch (final TimeoutException e)
                {
                    notifyTimeout(sequence.get());
                }
                catch (final AlertException ex)
                {
                    if (!running.get())
                    {
                        break;
                    }
                }
                catch (final Throwable ex)
                {
                    exceptionHandler.handleEventException(ex, nextSequence, event);
                    sequence.set(nextSequence);
                    nextSequence++;
                }
            }
        }
        finally
        {
            notifyShutdown();
            running.set(false);
        }
    }

有5个Handler对应5个BatchEventProcessor

4）、多生产者多消费者模型

依赖Disruptor配置实现多生产者：

    public Disruptor(final EventFactory<T> eventFactory,
                     final int ringBufferSize,
                     final Executor executor,
                     final ProducerType producerType,
                     final WaitStrategy waitStrategy)

producerType = ProducerType.MULTI

依赖WorkerPool实现多消费者：

    public WorkerPool(final RingBuffer<T> ringBuffer,
                      final SequenceBarrier sequenceBarrier,
                      final ExceptionHandler<? super T> exceptionHandler,
                      final WorkHandler<? super T>... workHandlers)

1）多消费者序号问题

每个消费者有一个sequence序号，用于单独统计消费进度

在这里插入图片描述

假设有2个生产者3个消费者，RingBuffer中有10条数据

消费者1消费第0、1条数据，消费者2消费第2、3、5条数据，消费者3消费第4条数据

在这里插入图片描述

消费者1消费完第0条数据，第1条数据未消费；消费者2消费完第2、3条数据，第5条数据未消费；消费者3消费完第4条数据

在这里插入图片描述

此时生产者要投递数据，会去循环三个消费者比较当前sequence最小的值，最小sequence值为1，所以生产者只能写入第0个槽，不能写入第1个槽，等到消费者1消费完第1条数据，再按照最小的进度（sequence=5）来判断是否可以写入数据

2）代码实现

@Data
public class Order {
    private String id;

    private String name;

    private double price;
}

生产者：

public class Producer {
    private RingBuffer<Order> ringBuffer;

    public Producer(RingBuffer<Order> ringBuffer) {
        this.ringBuffer = ringBuffer;
    }

    public void sendData(String data) {
        long sequence = ringBuffer.next();
        try {
            Order order = ringBuffer.get(sequence);
            order.setId(data);
        } finally {
            ringBuffer.publish(sequence);
        }
    }
}

消费者：

public class Consumer implements WorkHandler<Order> {
    private String consumerId;

    private static AtomicInteger count = new AtomicInteger(0);

    private Random random = new Random();

    public Consumer(String consumerId) {
        this.consumerId = consumerId;
    }

    @Override
    public void onEvent(Order event) throws Exception {
        Thread.sleep(random.nextInt(5));
        System.out.println("当前消费者:" + this.consumerId + ",消费信息id:" + event.getId());
        count.incrementAndGet();
    }

    public int getCount() {
        return count.get();
    }
}

主函数：

public class Main {
    public static void main(String[] args) throws InterruptedException {
        // 1.创建ringBuffer
        RingBuffer<Order> ringBuffer = RingBuffer.create(
                ProducerType.MULTI,
                new EventFactory<Order>() {
                    @Override
                    public Order newInstance() {
                        return new Order();
                    }
                },
                1024 * 1024,
                new YieldingWaitStrategy());
        // 2.通过ringBuffer创建一个屏障
        SequenceBarrier sequenceBarrier = ringBuffer.newBarrier();
        // 3.创建多个消费者数组
        Consumer[] consumers = new Consumer[10];
        for (int i = 0; i < consumers.length; ++i) {
            consumers[i] = new Consumer("c" + i);
        }
        // 4.构建多消费者工作池
        WorkerPool<Order> workerPool = new WorkerPool<>(
                ringBuffer,
                sequenceBarrier,
                new EventExceptionHandler(),
                consumers);
        // 5.设置多个消费者的sequence序号 用于单独统计消费进度,并且设置到ringBuffer中
        ringBuffer.addGatingSequences(workerPool.getWorkerSequences());
        // 6.启动workerPool
        workerPool
                .start(Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()));
        CountDownLatch latch = new CountDownLatch(1);
        for (int i = 0; i < 100; ++i) {
            Producer producer = new Producer(ringBuffer);
            new Thread(() -> {
                try {
                    latch.await();
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
                for (int j = 0; j < 100; ++j) {
                    producer.sendData(UUID.randomUUID().toString());
                }
            }).start();
        }
        TimeUnit.SECONDS.sleep(2);
        System.out.println("-----线程创建完毕,开始生产数据-----");
        latch.countDown();

        TimeUnit.SECONDS.sleep(10);
        System.out.println("任务总数:" + consumers[0].getCount());
    }

    static class EventExceptionHandler implements ExceptionHandler<Order> {

        @Override
        public void handleEventException(Throwable ex, long sequence, Order event) {

        }

        @Override
        public void handleOnStartException(Throwable ex) {

        }

        @Override
        public void handleOnShutdownException(Throwable ex) {

        }
    }
}

运行结果：

-----线程创建完毕,开始生产数据-----
当前消费者:c5,消费信息id:a824214a-2035-41f9-9ceb-9d963df5c073
当前消费者:c6,消费信息id:17db1281-c6e7-42d8-a266-d9bbb069d4ab
当前消费者:c9,消费信息id:0940dea9-7298-4fe4-83d9-bc5bbfb53a75
当前消费者:c9,消费信息id:c4351d84-746e-488b-876e-62c9e3c9a1df
当前消费者:c0,消费信息id:2e949f20-c065-48fd-a728-4ba414f1a5f9
当前消费者:c4,消费信息id:0316ee28-a027-44aa-9ecf-efba9fc6f8b8
...
任务总数:10000

3）单消费者由BatchEventProcessor处理，多消费者由WorkProcessor来处理

public final class WorkerPool<T>
{

    private final Sequence workSequence = new Sequence(Sequencer.INITIAL_CURSOR_VALUE);
    private final WorkProcessor<?>[] workProcessors;

    public WorkerPool(final RingBuffer<T> ringBuffer,
                      final SequenceBarrier sequenceBarrier,
                      final ExceptionHandler<? super T> exceptionHandler,
                      final WorkHandler<? super T>... workHandlers)
    {
        this.ringBuffer = ringBuffer;
      	// 每个消费者对应的一个WorkProcessor
        final int numWorkers = workHandlers.length;
        workProcessors = new WorkProcessor[numWorkers];
				
        for (int i = 0; i < numWorkers; i++)
        {
            workProcessors[i] = new WorkProcessor<T>(ringBuffer,
                                                     sequenceBarrier,
                                                     workHandlers[i],
                                                     exceptionHandler,
                                                     workSequence);
        }
    }

		public RingBuffer<T> start(final Executor executor)
    {
        if (!started.compareAndSet(false, true))
        {
            throw new IllegalStateException("WorkerPool has already been started and cannot be restarted until halted.");
        }

        final long cursor = ringBuffer.getCursor();
        workSequence.set(cursor);

        for (WorkProcessor<?> processor : workProcessors)
        {
            processor.getSequence().set(cursor);
            executor.execute(processor);
        }

        return ringBuffer;
    }

4、Disruptor核心原理

1）、为什么Disruptor性能这么好？

数据结构层面：使用环形结构、数组、内存预加载
使用单线程写方式、内存屏障
消除伪共享（填充缓存行）
序号栅栏和序号配合使用来消除锁和CAS

2）、数据结构层面

RingBuffer使用数组Object[] entries作为存储元素，如下图所示：

在这里插入图片描述

abstract class RingBufferFields<E> extends RingBufferPad
{
    private final Object[] entries;

    RingBufferFields(EventFactory<E> eventFactory,
                     Sequencer       sequencer)
    {
        this.sequencer  = sequencer;
        this.bufferSize = sequencer.getBufferSize();

        if (bufferSize < 1)
        {
            throw new IllegalArgumentException("bufferSize must not be less than 1");
        }
        if (Integer.bitCount(bufferSize) != 1)
        {
            throw new IllegalArgumentException("bufferSize must be a power of 2");
        }

        this.indexMask = bufferSize - 1;
        this.entries   = new Object[sequencer.getBufferSize() + 2 * BUFFER_PAD];
        fill(eventFactory);
    }

    /**
     * 内存预加载
     *
     * @param eventFactory
     */  
    private void fill(EventFactory<E> eventFactory)
    {
        for (int i = 0; i < bufferSize; i++)
        {
            entries[BUFFER_PAD + i] = eventFactory.newInstance();
        }
    }

3）使用单线程写

Disruptor的RingBuffer之所以可以做到完全无锁，也是因为单线程写

4）内存屏障

对应到Java语言，就是volatile变量与happens before语义

内存屏障：Linux的smp_wmb()/smp_rmb()

5）消除伪共享

缓存系统中是以缓存行（cache line）为单位存储的

缓存行是2的整数幂个连续字节，一般为32-256个字节

最常见的缓存行大小是64个字节

当多线程修改互相独立的变量时，如果这些变量共享同一个缓存行，就会无意中影响彼此的性能，这就是伪共享

class LhsPadding
{
    protected long p1, p2, p3, p4, p5, p6, p7;
}

class Value extends LhsPadding
{
    protected volatile long value;
}

class RhsPadding extends Value
{
    protected long p9, p10, p11, p12, p13, p14, p15;
}

public class Sequence extends RhsPadding
{

value左边填充7个long类型（8个字节），右边填充7个long类型，无论什么情况value永远占一个缓存行

6）序号栅栏机制

在生产者投递Event的时候，总是会使用：

        long sequence = ringBuffer.next();

Disruptor3.x中，序号栅栏SequenceBarrier和序号Sequence搭配使用，协调和管理消费者与生产者的工作节奏，避免了锁和CAS的使用

Disruptor3.x中，各个消费者和生产者持有自己的序号，这些序号的变化必须满足如下基本条件：

消费者序号数值必须小于生产者序号数值
消费者序号数值必须小于其前置（依赖关系）消费者的序号数值
生产者序号数值不能大于消费者中最小的序号数值，以避免生产者速度过快，将还没来得及消费的消息覆盖

public final class SingleProducerSequencer extends SingleProducerSequencerFields
{

    @Override
    public long next()
    {
        return next(1);
    }

    @Override
    public long next(int n)
    {
        if (n < 1)
        {
            throw new IllegalArgumentException("n must be > 0");
        }
		// nextValue初始值-1
        long nextValue = this.nextValue;
		// nextSequence=-1+1=0
        long nextSequence = nextValue + n;
      	// 假设bufferSize=10 wrapPoint=0-10=-10
        long wrapPoint = nextSequence - bufferSize;
        long cachedGatingSequence = this.cachedValue;

        if (wrapPoint > cachedGatingSequence || cachedGatingSequence > nextValue)
        {
            // 最小的序号
            long minSequence;
            // 自旋操作
            // Util.getMinimumSequence(gatingSequences, nextValue):找到消费者中最小的序号值
          	// 如果(生产者序号-bufferSize)大于消费者中最小的序号,那么就挂起进行自旋操作
          	// 其实就是判断上一圈的序号(也就是生产者序号-bufferSize)是否被消费完
            while (wrapPoint > (minSequence = Util.getMinimumSequence(gatingSequences, nextValue)))
            {
                LockSupport.parkNanos(1L);
            }
			// cachedValue接收了最小的消费者序号
            this.cachedValue = minSequence;
        }

        this.nextValue = nextSequence;

        return nextSequence;
    }

7）、WaitStrategy等待策略

public final class BlockingWaitStrategy implements WaitStrategy
{
    private final Lock lock = new ReentrantLock();
    private final Condition processorNotifyCondition = lock.newCondition();

    @Override
    public long waitFor(long sequence, Sequence cursorSequence, Sequence dependentSequence, SequenceBarrier barrier)
        throws AlertException, InterruptedException
    {
        long availableSequence;
       	// cursorSequence生产者序号
      	// 如果生产者序号小于消费序号下一个想要的序号(消费者没有可消费的数据),等待
        if ((availableSequence = cursorSequence.get()) < sequence)
        {
            lock.lock();
            try
            {
                while ((availableSequence = cursorSequence.get()) < sequence)
                {
                    barrier.checkAlert();
                    processorNotifyCondition.await();
                }
            }
            finally
            {
                lock.unlock();
            }
        }

        while ((availableSequence = dependentSequence.get()) < sequence)
        {
            barrier.checkAlert();
        }

        return availableSequence;
    }

    @Override
    public void signalAllWhenBlocking()
    {
        lock.lock();
        try
        {
            // 生产者投递数据时,唤醒所有等待状态的
            processorNotifyCondition.signalAll();
        }
        finally
        {
            lock.unlock();
        }
    }
}

8）、EventProcessor

public final class BatchEventProcessor<T>
    implements EventProcessor
{
  	
  	@Override
    public void run()
    {
        if (!running.compareAndSet(false, true))
        {
            throw new IllegalStateException("Thread is already running");
        }
        sequenceBarrier.clearAlert();

        notifyStart();

        T event = null;
      	// 消费者下一个想要的序号,假设是10
        long nextSequence = sequence.get() + 1L;
        try
        {
            while (true)
            {
                try
                {
                  	// availableSequence生产者序号,假设是13
                    final long availableSequence = sequenceBarrier.waitFor(nextSequence);
					// 循环把生产者序号前的数据都消费完
                    while (nextSequence <= availableSequence)
                    {
                        event = dataProvider.get(nextSequence);
                        eventHandler.onEvent(event, nextSequence, nextSequence == availableSequence);
                        nextSequence++;
                    }
					// 当前消费完的序号为availableSequence(生产者序号)
                    sequence.set(availableSequence);
                }
                catch (final TimeoutException e)
                {
                    notifyTimeout(sequence.get());
                }
                catch (final AlertException ex)
                {
                    if (!running.get())
                    {
                        break;
                    }
                }
                catch (final Throwable ex)
                {
                    exceptionHandler.handleEventException(ex, nextSequence, event);
                    sequence.set(nextSequence);
                    nextSequence++;
                }
            }
        }
        finally
        {
            notifyShutdown();
            running.set(false);
        }
    }

推荐资料：

高性能Java并发框架disruptor源码解析与实战

邋遢的流浪剑客

关注

2
点赞
踩
7

收藏

觉得还不错? 一键收藏
打赏
0
评论
Disruptor学习笔记：基本使用、核心概念和原理

Disruptor是英国外汇交易公司LMAX开发的一个高性能队列（系统内部的内存队列），研发的初衷是解决内存队列的延迟问题（在性能测试中发现竟然与I/O操作处于同样的数量级）。基于Disruptor开发的系统单线程能支撑每秒600万订单，2010年在QCon演讲后，获得了业界关注。2011年，企业应用软件专家Martin Fowler专门撰写长文介绍。同年它还获得了Oracle官方的Duke大奖1、快速入门1）、步骤创建一个工厂Event类，用于创建Event类实例对象需要有一个监听事件类，用于处
复制链接

扫一扫