循环Future异步调用简单探索

需求

最近需要将公司的一个微信服务号的关注用户数据从微信端取出返回一个insert-sql。

思路

通过调用微信公众号平台提供接口进行查询数据，但这个接口有个限制，每次查询的list.size<=100，所以我只能将参数进行100个切分查询最后将数据进行汇总
文档：https://developers.weixin.qq.com/doc/offiaccount/User_Management/Get_users_basic_information_UnionID.html#UinonId

开始

Future

首先我使用future来进行异步开发：这里没有完全自测

    public ResponseVO<Set<String>> getUnionIdToSQL(List<String> openIds,String token,@RequestParam String appId, String appSecret) {
        if (Objects.isNull(openIds)){
            return new ResponseVO<>(-1,"请传参数");
        }
        Long startTime = DateHelper.currentTimeMillis();
        Set<String> resultList = new ConcurrentHashSet();
        List<Future> futures = Lists.newArrayList(); 
        List<String> openIdList = Lists.newArrayList();
        int count = 0;
        for (int i = 0; i<= openIds.size();i++){
            if (count>0 && count%100 == 0 || i==openIds.size()){
                List<String> finalOpenIdList = openIdList;
                Future future = executorService.submit(() -> {
                    List<String> unionIdToSQL = userUnionService.getUnionIdToSQL(finalOpenIdList, token, appId, appSecret);
                    resultList.addAll(unionIdToSQL);
                });
                openIdList = Lists.newArrayList();
                future.get();
            }
            count ++;
            if (i<openIds.size()){
                openIdList.add(openIds.get(i));
            }
        }

        for(Future future : futures){
        //这里会有异常抛出，需要手动处理下
            future.get();
        }

        Long endTime = DateHelper.currentTimeMillis();
        log.info("getUnionIdToSQL cost time is : [{}]",endTime-startTime);
        return new ResponseVO<>(resultList);

    }

CompletableFuture实现

public ResponseVO<Set<String>> getUnionIdToSQL(List<String> openIds, String token,String appId,  String appSecret) {
        if (Objects.isNull(openIds)){
            return new ResponseVO<>(-1,"请传参数");
        }
        Long startTime = DateHelper.currentTimeMillis();
        Set<String> resultList = new ConcurrentHashSet();
        List<CompletableFuture<List<String>>> futures = Lists.newArrayList();
        List<String> openIdList = Lists.newArrayList();
        int count = 0;
        for (int i = 0; i<= openIds.size();i++){
            if (count>0 && count%100 == 0 || i==openIds.size()){
                List<String> finalOpenIdList = openIdList;
                futures.add(CompletableFuture.supplyAsync(
                        () -> userUnionService.getUnionIdToSQL(finalOpenIdList, token, appId, appSecret), executorService));
                openIdList = Lists.newArrayList();

            }
            count ++;
            if (i<openIds.size()){
                openIdList.add(openIds.get(i));
            }
        }
        futures.stream().forEach((future) -> {
        //这里的异常需要手动处理下
            resultList.addAll(future.get());
        });

        Long endTime = DateHelper.currentTimeMillis();
        log.info("getUnionIdToSQL cost time is : [{}]",endTime-startTime);
        return new ResponseVO<>(resultList);

    }

最后给大家看下我最后的修改版本，对异常进行了处理：

    public ResponseVO<Set<String>> getUnionIdToSQL(List<String> openIds,String token,@RequestParam String appId, String appSecret) {
        if (Objects.isNull(openIds)){
            return new ResponseVO<>(-1,"请传参数");
        }
        Long startTime = DateHelper.currentTimeMillis();
        Set<String> resultList = new ConcurrentHashSet();
        List<Future> futures = Lists.newArrayList(); 
        List<String> openIdList = Lists.newArrayList();
        int count = 0;
        for (int i = 0; i<= openIds.size();i++){
            if (count>0 && count%100 == 0 || i==openIds.size()){
                List<String> finalOpenIdList = openIdList;
                Future future = executorService.submit(() -> {
                    List<String> unionIdToSQL = userUnionService.getUnionIdToSQL(finalOpenIdList, token, appId, appSecret);
                    resultList.addAll(unionIdToSQL);
                });
                openIdList = Lists.newArrayList();
                future.get();
            }
            count ++;
            if (i<openIds.size()){
                openIdList.add(openIds.get(i));
            }
        }

        for(Future future : futures){
        //这里会有异常抛出，需要手动处理下
            future.get();
        }

        Long endTime = DateHelper.currentTimeMillis();
        log.info("getUnionIdToSQL cost time is : [{}]",endTime-startTime);
        return new ResponseVO<>(resultList);

    }

CompletableFuture实现

public ResponseVO<Set<String>> getUnionIdToSQL( List<String> openIds, String token, String appId,  String appSecret) {
        if (Objects.isNull(openIds)){
            return new ResponseVO<>(-1,"请传参数");
        }
        Long startTime = DateHelper.currentTimeMillis();
        Set<String> resultList = new ConcurrentHashSet();
        List<CompletableFuture<List<String>>> futures = Lists.newArrayList();
        List<String> openIdList = Lists.newArrayList();
        int count = 0;
        for (int i = 0; i<= openIds.size();i++){
            if (count>0 && count%100 == 0 || i==openIds.size()){
                List<String> finalOpenIdList = openIdList;
                futures.add(CompletableFuture.supplyAsync(
                        () -> userUnionService.getUnionIdToSQL(finalOpenIdList, token, appId, appSecret), executorService));
                openIdList = Lists.newArrayList();

            }
            count ++;
            if (i<openIds.size()){
                openIdList.add(openIds.get(i));
            }
        }
        futures.stream().forEach((future) -> {
            try{
                resultList.addAll(future.get());
            }catch (Throwable throwable){
                log.error("异步调用发生异常：" + throwable.getMessage());
            }
        });

        Long endTime = DateHelper.currentTimeMillis();
        log.info("getUnionIdToSQL cost time is : [{}]",endTime-startTime);
        return new ResponseVO<>(resultList);

    }

微信的具体调用处理逻辑，我就不展示了，这里只是探讨异步调用。
上面的切分list可以使用list的subList()方法，但是注意需要考虑最后的边界问题。

结束语

在这里没有考虑其他的算法切分，在这里只是通过该代码分享在循环异步调用时应该怎么处理，所以考虑上并没有太深入，因为我很懒，如果你看到哪里有什么可以改进的地方，请评论。