线程池+多线程获取大量数据方法以及批量更新

1.业务需求

在某些业务场景下面，需要大量的获取数据，但是请求只能一次获取2000条（比如广点通与某些广告接口AIP），如果写分页节后一页一页的获取数据，然后批量保存或更新会很慢。在这个问题下本文章解决思路是采用线程池+多线程获取数据的方法来解决时间过长的问题。

1. 设置自己的线程池大小 ，配置适合自己合适的参数；

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingDeque;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

/**
 * @Created by xinyu liang
 * @Data: 2021/12/15 16:15
 */
@Configuration
public class MyThreadConfig {

    @Bean
    public ThreadPoolExecutor threadPoolExecutor() {
        return new ThreadPoolExecutor(
                20,
                200,
                10,
                TimeUnit.SECONDS,
                new LinkedBlockingDeque<>(100000),
                Executors.defaultThreadFactory(),
                new ThreadPoolExecutor.AbortPolicy()
        );
    }
}

2. 在业务层将线程池导入，然后多线程任务交给线程池来处理 ；

 	//线程池
    @Resource
    private ThreadPoolExecutor executor;

2.代码

1. 业务层代码 ；

 @Override
    public R getAllData(Integer currentPage,Integer pageSize){
        List<StudentCustomer> totalData = new ArrayList<>();
        //任务开始时间
        long startTime = System.currentTimeMillis();
        Page<StudentCustomer> page = new Page<>(1,pageSize);
        IPage<StudentCustomer> studentCustomerIPage = baseMapper.selectPage(page, null);
        List<Page<StudentCustomer>> pages = new ArrayList<>();
        for (int i = 1; i <= studentCustomerIPage.getPages() ; i++){
            pages.add( new Page<>(i,pageSize));
        }
        /**
         * 方法执行完后端处理
         */
        CompletableFuture[] completableFutures = pages.stream().map(v ->
                CompletableFuture.supplyAsync(() -> {
                    IPage<StudentCustomer> iPage = this.page(v, null);
                    log.info("接收到分页数据是：{}", iPage.getRecords().size());
                    return iPage;
                }, executor).whenComplete((res, throwable) -> {
                    List<StudentCustomer> collect = res.getRecords().stream().map(v1 -> {
                        v1.setName(v1.getName() + "+1s");
                        return v1;
                    }).collect(Collectors.toList());
                    totalData.addAll(collect);
                })).toArray(CompletableFuture[]::new);
        CompletableFuture.allOf(completableFutures).join();
        //任务执行总时长
        long times = System.currentTimeMillis() - startTime;
        System.err.println("任务执行总时长" + times / 1000);
        return R.data("任务执行总时长" + times / 1000);
    }

3.分批量快速保存数据

1. 分割数据

private  List<List<StudentCustomer>> splitList(List<StudentCustomer> list, int len) {
        if (list == null || list.size() == 0 || len < 1) {
            return null;
        }
        List<List<StudentCustomer>> result = new ArrayList<List<StudentCustomer>>();
        int size = list.size();
        int count = (size + len - 1) / len;
        for (int i = 0; i < count; i++) {
            List<StudentCustomer> subList = list.subList(i * len, ((i + 1) * len > size ? size : len * (i + 1)));
            result.add(subList);
        }
        return result;
    }

2. 保存大量数据代码

//任务开始时间
        long startTime = System.currentTimeMillis();

        List<StudentCustomer> students = this.list(new LambdaQueryWrapper<>());
        List<StudentCustomer> collect = students.stream().map(v -> {
            v.setName(v.getName() + "|" + v.getId());
            return v;
        }).collect(Collectors.toList());
        List<Boolean> results = new ArrayList<>();
        List<List<StudentCustomer>> lists = this.splitList(collect, 100);

        CompletableFuture[] completableFutures = lists.stream().map(v ->
                CompletableFuture.supplyAsync(() -> {
                    boolean b = this.updateBatchById(v);
                    return b? b:false;
                }, executor).whenComplete((res, throwable) -> {
                    results.add(res);
                })).toArray(CompletableFuture[]::new);

        CompletableFuture.allOf(completableFutures).join();
        System.err.println(results);
        //任务执行总时长
        long times = System.currentTimeMillis() - startTime;
        System.err.println("任务执行总时长" + times / 1000);
        return R.data("任务执行总时长" + times / 1000);