多线程竞争消费 vs 一个管理者+一堆worker_多个线程竞争消费-CSDN博客

本文链接：https://blog.csdn.net/xiangyubobo/article/details/64920561

背景：最近在做日志收集，由于种种原因，我们放弃了logstash，采用自己写consumer。consumer结构很简单，三个部分，每个部分都用Linkedblockingqueue传递数据

日志接收线程—>日志处理线程池—>日志落地线程池

在最初的版本里，日志处理线程池就是简单的ExecutorService，每个线程不停的循环从队列里面取数据，然后业务处理，并提交给落地线程池。

// in main.java
// 线程池初始化
ExecutorService executorCon = Executors.newFixedThreadPool(5);
// 数据队列
LinkedBlockingQueue<String> queue = new LinkedBlockingQueue<String>(20000);
List<Consumer> consumers = Lists.newArrayList();
// 产生5个消费线程并开始工作
for (int i = 0; i < 5; ++i) {
    Consumer c = new Consumer(queue);
    executorCon.submit(c);
    consumers.add(c);
}

// in Consumer.java
// 线程运行的逻辑
public void run() {
    while (running) {
        try {
            // 从队列中取数
            String data = queue.poll(POLL_TIME, TimeUnit.MILLISECONDS);
            if (data != null) {
                // 处理数据
                dealData(data);
            } else {
                LOGGER.warn("not polled data");
            }
        } catch (Exception e) {
            LOGGER.error("poll data exception! ", e);
        }
    }

    // 任务结束时候的收尾工作
    String data = null;
    try {
        while ((data = queue.poll(POLL_TIME, TimeUnit.MILLISECONDS)) != null) {
            dealData(data);
        }
    } catch (InterruptedException e) {
        LOGGER.error("", e);
    }
    LOGGER.info("end consumer");
}

但线上运行发现随着日志处理线程数量的增加，系统的吞吐并没有增加很多。而且有时候会出现数据队列被塞满了的情况。于是有同学提出了一种改版，用一个线程管理，它负责从数据队列中取数，然后将多条日志组装成一个任务；其他线程和以前的消费线程一样的处理逻辑。

// in DealManager.java
private ExecutorService service = Executors.newFixedThreadPool(1);
private ExecutorService executor = Executors.newFixedThreadPool(4);
private static final int POLL_TIME = 50;
private static int BATCH_DATA_PER_TASK = 100;

private void patchTask() {
    try {
        if (currentTask == null) {
            currentTask = new Task();
        }
        // 批量取日志，然后加入到当前任务中
        for (int i = 0; i < BATCH_DATA_PER_TASK; ++i) {
            String data = queue.poll(POLL_TIME, TimeUnit.MILLISECONDS);
            if (data != null) {
                currentTask.addData(data);
            } else {
                LOGGER.warn("not polled data");
                break;
            }
        }
        // 如果凑齐“一批”日志，那么认为任务填充完毕，交给线程池处理
        if (currentTask.getDataCount() >= 100) {
            executor.submit(currentTask);
            currentTask = null;
        }
    } catch (InterruptedException e) {
        LOGGER.error("", e);
    }
}

public void run() {
    while (running) {
        patchTask();
    }
}

每一个task都是一个runnable的任务，由DealManager里面的线程池去执行它们

// in Task.java
public static class Task implements Runnable {

    private List<String> datas = Lists.newArrayList();

    private StringBuffer stringBuffer = new StringBuffer(512);

    public void addData(String data) {
        datas.add(data);
    }

    public int getDataCount() {
        return datas.size();
    }

    public void run() {
        for (String data : datas) {
            // 没有实际含义，只是为了做一些操作，模拟业务上对日志的处理
            // 这里的处理和前面日志处理线程中的 dealData 函数中一样
            Random ran = new Random(System.currentTimeMillis());
            for (int count = 0; count < 50; ++count) {
                for (int i = 0; i < 10; ++i) {
                    String part0 = data.substring(i * 10, i * 10 + 10);
                    for (int j = 0; j < part0.length(); ++j) {
                        int seed = 26 - (part0.charAt(j) - '0');
                        stringBuffer.append(ran.nextInt(seed));
                    }
                    stringBuffer.delete(0, stringBuffer.length());
                }
            }
        }
        LOGGER.info("end deal task");
    }
}

为了模拟真实环境，写了一个产生随机文本的生产者

// in Productor.java
public class Productor implements Callable<Boolean> {

    private static final Logger LOGGER = LoggerFactory.getLogger(Productor.class);

    private static final int STRING_SIZE = 512;

    private static final int SLEEP_TIME = 15;

    private StringBuilder stringBuilder;

    private LinkedBlockingQueue<String> queue;

    public Productor(LinkedBlockingQueue<String> queue) {
        this.queue = queue;
        this.stringBuilder = new StringBuilder(STRING_SIZE);
    }

    public void run() {

    }

    public Boolean call() throws Exception {
        LOGGER.info("start productor");
        for (int count = 0 ; count < 100000; ++count) {
            // 产生长度固定为512的随机文本，作为一行日志
            Random ran = new Random(System.currentTimeMillis());
            for (int i = 0; i < STRING_SIZE; ++i) {
                stringBuilder.append(ran.nextInt());
            }
            try {
                // 尝试塞入数据队列中
                if (queue.offer(stringBuilder.toString(), 20, TimeUnit.MILLISECONDS)) {

                } else {
                    LOGGER.warn("queue is full!");
                    Thread.sleep(SLEEP_TIME);
                }
            } catch (Exception e) {
                LOGGER.error("insert string failed! ", e);
            }
            stringBuilder.delete(0, stringBuilder.length());
        }
        LOGGER.info("end productor");
        // 生产完毕返回上层，以便上层通知消费线程停止消费
        return true;
    }
}

经过实验，在相同的生产者，数据队列大小（2w）和消费线程数下（5），生产者连续产生10w条随机的文本，第二种方案的吞吐比第一种高，实验数据如下

一个生产者，一个管理，4个消费
18:35:00.258 [pool-1-thread-1] INFO  com.baidu.xyb.Producer - start productor
18:35:19.634 [pool-1-thread-1] INFO  com.baidu.xyb.Producer - end productor
18:35:19.635 [main] INFO  com.baidu.xyb.DealManager - stop deal-manager
18:35:25.204 [main] INFO  com.baidu.xyb.DealManager - stop poll
18:35:28.191 [main] INFO  com.baidu.xyb.DealManager - end deal-manager
cost:28207

一个生产者，5个消费
18:33:49.017 [pool-1-thread-1] INFO  com.baidu.xyb.Producer - start productor
18:34:03.507 [pool-1-thread-1] WARN  com.baidu.xyb.Producer - queue is full!
18:34:12.398 [pool-1-thread-1] WARN  com.baidu.xyb.Producer - queue is full!
18:34:14.123 [pool-1-thread-1] WARN  com.baidu.xyb.Producer - queue is full!
18:34:20.447 [pool-1-thread-1] WARN  com.baidu.xyb.Producer - queue is full!
18:34:23.318 [pool-1-thread-1] INFO  com.baidu.xyb.Producer - end productor
18:34:23.318 [main] INFO  com.baidu.xyb.Consumer - stop consumer
18:34:23.319 [main] INFO  com.baidu.xyb.Consumer - stop consumer
18:34:23.319 [main] INFO  com.baidu.xyb.Consumer - stop consumer
18:34:23.319 [main] INFO  com.baidu.xyb.Consumer - stop consumer
18:34:23.319 [main] INFO  com.baidu.xyb.Consumer - stop consumer
18:34:29.283 [pool-2-thread-3] INFO  com.baidu.xyb.Consumer - end consumer
18:34:29.290 [pool-2-thread-5] INFO  com.baidu.xyb.Consumer - end consumer
18:34:29.307 [pool-2-thread-4] INFO  com.baidu.xyb.Consumer - end consumer
18:34:29.292 [pool-2-thread-1] INFO  com.baidu.xyb.Consumer - end consumer
18:34:29.317 [pool-2-thread-2] INFO  com.baidu.xyb.Consumer - end consumer
cost:40559