Java多线程ThreadPoolTaskExecutor、CountDownLatch并发下载文件并打包zip

多线程并发下载文件

1.启用多线程下载文件，线程交给线程池调度，

2.在子线程里下载文件并写入zip  注意在并发写入时要对zip加锁

3.使用CountDownLatch保证主线程在所有子线程执行结束后再结束

 

zip打包使用的是 org.apache.commons:commons-compress:1.18

gradle:  compile group: 'org.apache.commons', name: 'commons-compress', version: '1.18'

maven:

<!-- https://mvnrepository.com/artifact/org.apache.commons/commons-compress -->
<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-compress</artifactId>
    <version>1.18</version>
</dependency>

 

上代码

首先线程池的配置类

IO类的线程池线程数建议 CPU核数*2+1

@Configuration
public class BeanConfig {

    @Value("${threadPoolTaskExecutor.corePoolSize:10}")
    private int corePoolSize;
    @Value("${threadPoolTaskExecutor.maxPoolSize:20}")
    private int maxPoolSize;
    @Value("${threadPoolTaskExecutor.queueCapacity:1000}")
    private int queueCapacity;
    @Value("${threadPoolTaskExecutor.keepAliveSeconds:300}")
    private int keepAliveSeconds;

    @Bean
    public ThreadPoolTaskExecutor taskExecutor() {
        ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
        //核心线程数
        taskExecutor.setCorePoolSize(corePoolSize);
        //最大线程数
        taskExecutor.setMaxPoolSize(maxPoolSize);
        //队列长度
        taskExecutor.setQueueCapacity(queueCapacity);
        taskExecutor.setKeepAliveSeconds(keepAliveSeconds);
        taskExecutor.initialize();
        return taskExecutor;
    }
}

然后时用来实现下载功能的线程task类 实现Callable

public class DownloadTask implements Callable {

    private final static Logger logger = LoggerFactory.getLogger(DownloadTask.class);

    //图片的详情  包括名称  url 等详细信息
    private List<Data> dataList;

    private CountDownLatch countDownLatch;

    private RestTemplate restTemplate;

    //zip打包类
    private ZipArchiveOutputStream zous;


    public DownloadTask(List<Data> dataList, CountDownLatch countDownLatch, RestTemplate restTemplate, ZipArchiveOutputStream zous) {
        this.dataList = dataList;
        this.countDownLatch = countDownLatch;
        this.restTemplate = restTemplate;
        this.zous = zous;
    }

    @Override
    public Object call() throws Exception {

        logger.info("start----------------------------" + Thread.currentThread().getName());

        try {

            for (Data data : dataList) {
                //从URL读取图片
                ResponseEntity<Resource> entity = restTemplate.getForEntity(data.getStorageRefFile(), Resource.class);
                InputStream inputStream = entity.getBody().getInputStream();
                //文件流转为byte数组
                byte[] bytes = FileUtil.getBytesFromInputStream(inputStream);

                String dir = data.getExternalNumber();
                String fileName = data.getOriginalFileName();

                if (StringsExpand.isNotEmpty(dir)) {
                    fileName = dir + File.separator + fileName;
                }
                // 压缩文件 写入zip
                // 加锁
                synchronized (zous){
                    ArchiveEntry entry = new ZipArchiveEntry(fileName);
                    zous.putArchiveEntry(entry);
                    zous.write(bytes);
                    zous.closeArchiveEntry();
                }

            }
            logger.info("end----------------------------" + Thread.currentThread().getName());
        } catch (Exception e) {
            e.printStackTrace();
            logger.error("批量下载影像失败",e);
        } finally {
            countDownLatch.countDown();
        }

        return null;
    }
}

通过线程池多线程下载的service方法

    @Autowired
    private ThreadPoolTaskExecutor threadPoolTaskExecutor;

public void downloadPipeLineImages(List<String> applyIds, HttpServletResponse response) throws Exception {

        logger.info("影像下载开始----------------------");

        long start = System.currentTimeMillis();

        String downloadName = "影像批量下载-" + DateUtil.getTodayInt() + ".zip";

        response.setContentType("multipart/form-data");
        response.setHeader("Content-Disposition", "attachment;fileName=" + URLEncoder.encode(downloadName, "UTF-8"));

        OutputStream outputStream = response.getOutputStream();

        ZipArchiveOutputStream zous = new ZipArchiveOutputStream(outputStream);
        //防止中文乱码
        zous.setEncoding(StandardCharsets.UTF_8.name());
        zous.setUseZip64(Zip64Mode.AsNeeded);

        //查询分类 categoryId
        //List<String> categoryIdList = 

        CountDownLatch countDownLatch = new CountDownLatch(applyIds.size());

        try {

            for (String applyId : applyIds) {

//                List<Data> dataList=getImageData(applyId,categoryIdList);
                //获取每个applyId的图片信息 包括url  文件名 等等
                List<Data> dataList = getTestData(applyId);
                if (StringsExpand.isNotEmpty(dataList)){
                    threadPoolTaskExecutor.submit(new DownloadTask(dataList, countDownLatch, restTemplate,zous));
                }
            }

            countDownLatch.await();
            long end = System.currentTimeMillis();
            logger.info("耗时================== " + (end - start)+ "ms");

        } catch (Exception e) {
            logger.error("文件打包下载出错", e);
            throw new CommException("文件打包下载出错");
        } finally {
            zous.close();
            logger.info("影像下载结束----------------------");
        }


    }