系列文章目录
![](https://img-blog.csdnimg.cn/direct/3e72ee911695497ca4915ddec2f4561d.png)
前言
在这个充满技术创新的时代,AI大模型正成为开发者们的新宠。它们可以帮助我们完成从简单的问答到复杂的编程任务,所以AI编程将会是未来的主流方向,利用AI大模型的能力,本文将介绍从零到一用AI大模型编写一个多线程并发框架。
一、项目背景
由于博主每次批量处理数据任务时,都需要去考虑线程池各种参数,例如考虑任务执行速率、任务监控、任务回调、异常处理等。由于每个需求的实际诉求不一样,所以数据处理任务的代替代码都一样,只有部分有差异,从而导致代码仓库里各种复制粘贴,
这不是博主想要的,于是,萌发了一个封装一个多线程组件框架的想法,只需要传入自定义任务等参数,其它东西都不需要关心。
使用示例如下:源码地址
// 创建一个MmcTaskExecutor实例,用于执行单次长耗时任务
// 下面是创建一个计算从1加到100的任务,总共100个任务,采用fork分治算法,阈值为10,总共任务为100 / 10 * 2 = 20个大任务,执行速率约为10/s
MmcTaskExecutor<Integer, Integer> mmcTaskExecutor = MmcTaskExecutor.<Integer, Integer>builder()
.taskSource(IntStream.rangeClosed(1, 100).boxed().collect(Collectors.toList())) // 设置任务源
.taskProcessor(x -> x.stream().reduce(0, Integer::sum)) // 设置任务处理方法
.taskMerger(Integer::sum) // 设置结果处理方法(可选)
.threshold(10) // 设置任务处理阈值(可选)
.taskName("mmcTaskExample") // 设置任务名称
.rateLimiter(10, 20) // 设置速率限制,容量为10,每秒产生令牌为20,休眠时间为10ms
.forkJoinPoolConcurrency(4) // 设置ForkJoinPool的并发度为4
.build();
// 同步执行并打印结果
System.out.println("result: " + mmcTaskExecutor.execute());
// 任务执行过程监控
[mmcTaskExample] Tasks submitted. Total tasks: 100
[mmcTaskExample] Task started. Completed tasks: 14, remaining tasks: 86
[mmcTaskExample] Task started. Completed tasks: 14, remaining tasks: 86
[mmcTaskExample] Task started. Completed tasks: 20, remaining tasks: 80
[mmcTaskExample] Task started. Completed tasks: 26, remaining tasks: 74
[mmcTaskExample] Task started. Completed tasks: 33, remaining tasks: 67
[mmcTaskExample] Task started. Completed tasks: 40, remaining tasks: 60
[mmcTaskExample] Task started. Completed tasks: 46, remaining tasks: 54
[mmcTaskExample] Task started. Completed tasks: 52, remaining tasks: 48
[mmcTaskExample] Task started. Completed tasks: 58, remaining tasks: 42
[mmcTaskExample] Task started. Completed tasks: 64, remaining tasks: 36
[mmcTaskExample] Task started. Completed tasks: 70, remaining tasks: 30
[mmcTaskExample] Task started. Completed tasks: 76, remaining tasks: 24
[mmcTaskExample] Task started. Completed tasks: 82, remaining tasks: 18
[mmcTaskExample] Task started. Completed tasks: 88, remaining tasks: 12
[mmcTaskExample] Task started. Completed tasks: 94, remaining tasks: 6
[mmcTaskExample] Task started. Completed tasks: 100, remaining tasks: 0
[mmcTaskExample] Tasks completed. Elapsed time: 1107 ms
result: 5050
二、第一轮对话-让AI大模型理解我们的诉求
1、首先告诉AI大模型我的想法和目的,让它充当编程专家,给出初始化代码。
2、下面是AI给的回复。
- 【首先,我们需要创建一个核心类MmcTask,它将实现RecursiveTask接口,并实现compute方法,用于执行真正的任务处理方法。】
import java.util.List;
import java.util.concurrent.RecursiveTask;
public class MmcTask<T, R> extends RecursiveTask<R> {
private List<T> taskSource;
private MmcTaskHandler<T, R> taskHandler;
private int threshold;
private int start;
private int end;
public MmcTask(List<T> taskSource, MmcTaskHandler<T, R> taskHandler, int threshold, int start, int end) {
this.taskSource = taskSource;
this.taskHandler = taskHandler;
this.threshold = threshold;
this.start = start;
this.end = end;
}
@Override
protected R compute() {
// 当任务数量小于等于阈值时,执行真正的任务处理方法
if (end - start <= threshold) {
return taskHandler.handle(taskSource.subList(start, end));
}
// 将任务拆分为两部分,并递归执行
int middle = (start + end) / 2;
MmcTask<T, R> leftTask = new MmcTask<>(taskSource, taskHandler, threshold, start, middle);
MmcTask<T, R> rightTask = new MmcTask<>(taskSource, taskHandler, threshold, middle, end);
leftTask.fork();
R rightResult = rightTask.compute();
R leftResult = leftTask.join();
// 合并两个任务的结果
return taskHandler.merge(leftResult, rightResult);
}
}
- 【接下来,我们需要创建一个MmcTaskHandler接口,用于处理任务并合并任务结果。】
public interface MmcTaskHandler<T, R> {
R handle(List<T> taskSource);
R merge(R leftResult, R rightResult);
}
- 【然后,我们创建一个MmcTaskExecutor类,用于构建任务容器并执行任务。】
import java.util.List;
import java.util.concurrent.ForkJoinPool;
public class MmcTaskExecutor<T, R> {
private List<T> taskSource;
private MmcTaskHandler<T, R> taskHandler;
private int threshold;
private ForkJoinPool forkJoinPool;
private MmcTaskExecutor(Builder<T, R> builder) {
this.taskSource = builder.taskSource;
this.taskHandler = builder.taskHandler;
this.threshold = builder.threshold;
this.forkJoinPool = builder.forkJoinPool;
}
public static <T, R> Builder<T, R> builder() {
return new Builder<>();
}
public R execute() {
MmcTask<T, R> mmcTask = new MmcTask<>(taskSource, taskHandler, threshold, 0, taskSource.size());
return forkJoinPool.invoke(mmcTask);
}
public static class Builder<T, R> {
private List<T> taskSource;
private MmcTaskHandler<T, R> taskHandler;
private int threshold = 10;
private ForkJoinPool forkJoinPool = new ForkJoinPool();
public Builder<T, R> taskSource(List<T> taskSource) {
this.taskSource = taskSource;
return this;
}
public Builder<T, R> taskHandler(MmcTaskHandler<T, R> taskHandler) {
this.taskHandler = taskHandler;
return this;
}
public Builder<T, R> threshold(int threshold) {
this.threshold = threshold;
return this;
}
public Builder<T, R> forkJoinPool(ForkJoinPool forkJoinPool) {
this.forkJoinPool = forkJoinPool;
return this;
}
public MmcTaskExecutor<T, R> build() {
return new MmcTaskExecutor<>(this);
}
}
}
- 【最后,我们编写一个示例,计算从1加到100的结果。】
import java.util.ArrayList;
import java.util.List;
public class MmcTaskExample {
public static void main(String[] args) {
List<Integer> taskSource = new ArrayList<>();
for (int i = 1; i <= 100; i++) {
taskSource.add(i);
}
MmcTaskHandler<Integer, Integer> taskHandler = new MmcTaskHandler<Integer, Integer>() {
@Override
public Integer handle(List<Integer> taskSource) {
int sum = 0;
for (Integer num : taskSource) {
sum += num;
}
return sum;
}
@Override
public Integer merge(Integer leftResult, Integer rightResult) {
return leftResult + rightResult;
}
};
MmcTaskExecutor<Integer, Integer> mmcTaskExecutor = MmcTaskExecutor.<Integer, Integer>builder()
.taskSource(taskSource)
.taskHandler(taskHandler)
.threshold(10)
.build();
Integer result = mmcTaskExecutor.execute();
System.out.println(result); // 输出5050
}
}
从AI大模型给出的代码来看,它基本能理解我们的意图,不过也存在一下问题:
- 我希望任务处理方法和结果处理方法可以分开传入,而AI给出的MmcTaskHandler是合并一起的;
- 我要求代码需要有注释,但AI实际上并没有给太多注释;
不过问题不大,我们可以通过多轮对话去优化。
二、第二轮对话-优化任务处理方法和结果处理方法
1、MmcTaskHandler应该要拆分,也就是MmcTaskExecutor.builder要支持传入真正处理任务的方法,也要支持传入merge结果的处理方法,因为有些任务可能是没有计算结果的,所以merge结果处理方法可以为空。
2、下面是AI大模型给的回复。
- 【根据您的要求,我们可以将MmcTaskHandler拆分为两个接口:MmcTaskProcessor用于处理任务,MmcTaskMerger用于合并任务结果。同时,我们修改MmcTaskExecutor.Builder类以支持传入这两个接口。首先,我们创建一个MmcTaskProcessor接口。】
public interface MmcTaskProcessor<T, R> {
R process(List<T> taskSource);
}
- 【接下来,我们创建一个MmcTaskMerger接口。】
public interface MmcTaskMerger<R> {
R merge(R leftResult, R rightResult);
}
- 【然后,我们修改MmcTaskExecutor类。】
import java.util.List;
import java.util.concurrent.ForkJoinPool;
public class MmcTaskExecutor<T, R> {
private List<T> taskSource;
private MmcTaskProcessor<T, R> taskProcessor;
private MmcTaskMerger<R> taskMerger;
private int threshold;
private ForkJoinPool forkJoinPool;
private MmcTaskExecutor(Builder<T, R> builder) {
this.taskSource = builder.taskSource;
this.taskProcessor = builder.taskProcessor;
this.taskMerger = builder.taskMerger;
this.threshold = builder.threshold;
this.forkJoinPool = builder.forkJoinPool;
}
public static <T, R> Builder<T, R> builder() {
return new Builder<>();
}
public R execute() {
MmcTask<T, R> mmcTask = new MmcTask<>(taskSource, taskProcessor, taskMerger, threshold, 0, taskSource.size());
return forkJoinPool.invoke(mmcTask);
}
public static class Builder<T, R> {
private List<T> taskSource;
private MmcTaskProcessor<T, R> taskProcessor;
private MmcTaskMerger<R> taskMerger;
private int threshold = 10;
private ForkJoinPool forkJoinPool = new ForkJoinPool();
public Builder<T, R> taskSource(List<T> taskSource) {
this.taskSource = taskSource;
return this;
}
public Builder<T, R> taskProcessor(MmcTaskProcessor<T, R> taskProcessor) {
this.taskProcessor = taskProcessor;
return this;
}
public Builder<T, R> taskMerger(MmcTaskMerger<R> taskMerger) {
this.taskMerger = taskMerger;
return this;
}
public Builder<T, R> threshold(int threshold) {
this.threshold = threshold;
return this;
}
public Builder<T, R> forkJoinPool(ForkJoinPool forkJoinPool) {
this.forkJoinPool = forkJoinPool;
return this;
}
public MmcTaskExecutor<T, R> build() {
return new MmcTaskExecutor<>(this);
}
}
}
- 【最后,我们需要修改MmcTask类以适应这些更改。】
import java.util.List;
import java.util.concurrent.RecursiveTask;
public class MmcTask<T, R> extends RecursiveTask<R> {
private List<T> taskSource;
private MmcTaskProcessor<T, R> taskProcessor;
private MmcTaskMerger<R> taskMerger;
private int threshold;
private int start;
private int end;
public MmcTask(List<T> taskSource, MmcTaskProcessor<T, R> taskProcessor, MmcTaskMerger<R> taskMerger, int threshold, int start, int end) {
this.taskSource = taskSource;
this.taskProcessor = taskProcessor;
this.taskMerger = taskMerger;
this.threshold = threshold;
this.start = start;
this.end = end;
}
@Override
protected R compute() {
if (end - start <= threshold) {
return taskProcessor.process(taskSource.subList(start, end));
}
int middle = (start + end) / 2;
MmcTask<T, R> leftTask = new MmcTask<>(taskSource, taskProcessor, taskMerger, threshold, start, middle);
MmcTask<T, R> rightTask = new MmcTask<>(taskSource, taskProcessor, taskMerger, threshold, middle, end);
leftTask.fork();
R rightResult = rightTask.compute();
R leftResult = leftTask.join();
if (taskMerger != null) {
return taskMerger.merge(leftResult, rightResult);
} else {
return null;
}
}
}
- 【计算1到100的最终实例代码。】
import java.util.ArrayList;
import java.util.List;
public class MmcTaskExample {
public static void main(String[] args) {
List<Integer> taskSource = new ArrayList<>();
for (int i = 1; i <= 100; i++) {
taskSource.add(i);
}
MmcTaskProcessor<Integer, Integer> taskProcessor = new MmcTaskProcessor<Integer, Integer>() {
@Override
public Integer process(List<Integer> taskSource) {
int sum = 0;
for (Integer num : taskSource) {
sum += num;
}
return sum;
}
};
MmcTaskMerger<Integer> taskMerger = new MmcTaskMerger<Integer>() {
@Override
public Integer merge(Integer leftResult, Integer rightResult) {
return leftResult + rightResult;
}
};
MmcTaskExecutor<Integer, Integer> mmcTaskExecutor = MmcTaskExecutor.<Integer, Integer>builder()
.taskSource(taskSource)
.taskProcessor(taskProcessor)
.taskMerger(taskMerger)
.threshold(10)
.build();
Integer result = mmcTaskExecutor.execute();
System.out.println(result); // 输出5050
}
}
下一篇,继续优化这个多线程框架。。。
三、参考文章
- 《AI大模型编写多线程并发框架(六十一):从零开始搭建框架》
- 《AI大模型编写多线程并发框架(六十二):限流和并发度优化》
- 《AI大模型编写多线程并发框架(六十三):监听器优化》
- 《AI大模型编写多线程并发框架(六十四):发布和应用》
加我加群(备注csdn)一起交流学习!更多干货下载、项目源码和大厂内推等着你
![]() | ![]() |