双11活动数据导入

前言

毕业后，效力于创业电商公司，转正后的第一个大任务就是活动数据导入。众所周知，作为电商公司，要想发展壮大，肯定是需要在一年中举办各种各样的活动，才能吸引更多的用户购买，双11自从被BAT公司运用并尝到甜头后，各个电商公司也在进行着类似活动的举办，我亲历这次活动，感触挺多的，在此记录下来，为自己的成长刻上一个印记。

角色定义

• 开发（前端、后端）
• 产品（寻找需求，并定义需求）
• 类目运营（联系供应商）
• 活动运营（活动会场搭建，商品内容录入，维护）
• 摄制组（视频，图片美化）

工作流程（第一版活动）

1. 产品寻找到并定义需求（活动），产出PRD
2. 开需求研讨会，讨论功能实现的可能性及时间花费
3. 双线同行：用户端系统开发 & 运营联系供应商报名参加活动
4. 系统开发完毕，临时提供活动excel导入入口，将运营收集到的提报数据导入活动系统
5. 再从系统中导出一份给运营确认，确认无误后，运营制作活动排期excel
6. 活动排期excel导入活动系统后，生效，并开始售卖
   

系统开发步骤

用户端开发（h5）--》boss系统开发--》供应商平台开发

涉及技术

• POI解析excel
• Sequel Pro（Mysql客户端）
• OSS文件服务器（阿里云服务器）

活动系统导入流程剖析

说明：对于分布式系统，文字描述可能会更费力，不如直接上图。需要说明的是，4、5、6这三个场景是一样的，对于5、6不再贴图。

• 1、提报排期excel导入
  
• 2、生效价格配置并同步闪购件数到redis
  
• 3、设置活动商品标签
  
• 4、定时任务生效活动库存
  
• 5、定时任务活动内商品SKU全部上架售卖
• 6、定时任务商品SPU参加活动，该SPU下未参加活动的SKU下架
  #踩坑记录
• 1、解析excel问题：表头格式不正确、行记录包含特殊字符无法转换为数值型、权限不足无法读取文件
• 2、dubbo service调用批量接口，入参个数和返回值个数不一致，读取时判断
• 3、dubbo service调用超时

调用异步化，改成使用多线程循环调用dubbo service接口

经典代码摘抄

• 批量同步redis

            Pipeline pipeline = jedis.pipelined();
            spuScheduleList.forEach(spuSchedule ->{
                pipeline.hset(FlashSaleRedisHelper.getFlashSaleRedisKey(PromotionConstants.promotionId), String.valueOf(spuSchedule.getSpuId()), String.valueOf(spuSchedule.getCountLimit()));
            });
            pipeline.sync();

• dubbo service接口批量调用失败弥补

public List<PromotionError> batchSetPriceSchemeIfFailedThenTrySetBySingle(List<PriceSchemeConfigRequest> priceSchemeConfigList) {
        long start = System.currentTimeMillis();
        List<PromotionError> errorList = new ArrayList<>();
        try {
            innerPriceService.batchSetPriceScheme(priceSchemeConfigList);
        } catch (Exception e) {
            for (int i = 0; i < priceSchemeConfigList.size(); i++) {
                try {
                    innerPriceService.batchSetPriceScheme(priceSchemeConfigList.subList(i, i + 1));
                    logger.warn("单独设置价格配置数据：priceSchemeConfigList:{}", JSON.toJSONString(priceSchemeConfigList.get(i)));
                } catch (Exception sub) {
                    PriceSchemeConfigRequest priceSchemeConfigRequest = priceSchemeConfigList.get(i);
                    PromotionError promotionError = new PromotionError(0, priceSchemeConfigRequest.skuId, "设置价格方案失败");
                    errorList.add(promotionError);
                    logger.error(promotionError.toString());
                }
            }
        }
        logger.info("effectPriceScheme  getPriceSchemeIfFailedThenTryGetSingle size:{},cost:{}", priceSchemeConfigList.size(), System.currentTimeMillis() - start);
        return errorList;
    }

• 执行异步化 1、声明线程类

public class ExecuteSetPriceSchemeThread implements Callable<SubmitResult> {

    private PromotionInfo promotionInfo;
    private List<SkuInfoRequest> skuInfoRequestList;
    private ContractClient contractClient;
    private InnerPriceClient innerPriceClient;
    private static final Logger logger = LoggerFactory.getLogger(ExecuteSetPriceSchemeThread.class);

    public ExecuteSetPriceSchemeThread(List<SkuInfoRequest> skuInfoRequestList,
                                       PromotionInfo promotionInfo,
                                       ContractClient contractClient,
                                       InnerPriceClient innerPriceClient) {
        this.promotionInfo = promotionInfo;
        this.skuInfoRequestList = skuInfoRequestList;
        this.contractClient = contractClient;
        this.innerPriceClient = innerPriceClient;
    }

    @Override
    public SubmitResult call() throws Exception {
        long startMS = System.currentTimeMillis();
        SubmitResult submitResult = new SubmitResult();

        BigDecimal totalSize = BigDecimal.valueOf(skuInfoRequestList.size());
        int loopSize = totalSize.divide(innerBatchSize, BigDecimal.ROUND_UP).intValue();
        for (int index = 0; index < loopSize; index++) {
            int start = index * innerBatchSize.intValue();
            int end = (index + 1) * innerBatchSize.intValue();
            if (end > totalSize.intValue()) {
                end = totalSize.intValue();
            }
            List<SkuInfoRequest> skuInfoRequestForReal = skuInfoRequestList.subList(start, end);
            Map<Integer, PriceSchemeDetail> skuGeneralPriceSchemeMap = contractClient.getPriceSchemeIfFailedThenTryGetSingle(skuInfoRequestForReal);
            List<PriceSchemeConfigRequest> priceSchemeConfigList = buildPriceSchemeConfigRequestList(promotionInfo.promotionSkuMap,
                skuGeneralPriceSchemeMap,
                promotionInfo.activityPromotion,
                skuInfoRequestForReal,
                submitResult);
            List<PromotionError> errorList = innerPriceClient.batchSetPriceSchemeIfFailedThenTrySetBySingle(priceSchemeConfigList);
            submitResult.errorList.addAll(errorList);
        }
        logger.info("effectPriceScheme  skuInfoRequestList size:{},cost:{}", skuInfoRequestList.size(), System.currentTimeMillis() - startMS);
        return submitResult;
    }

    private List<PriceSchemeConfigRequest> buildPriceSchemeConfigRequestList(
        Map<Integer, ActivityProductPromotionSkuDTO> promotionSkuMap,
        Map<Integer, PriceSchemeDetail> skuGeneralPriceSchemeMap,
        ActivityProductPromotionDTO activityPromotion, List<SkuInfoRequest> skuInfoRequestForReal,
        SubmitResult submitResult) {
        List<PriceSchemeConfigRequest> priceSchemeConfigList = new ArrayList<>();
        skuInfoRequestForReal.forEach(skuInfoRequest -> {
            if (skuGeneralPriceSchemeMap.get(skuInfoRequest.skuId) == null) {
                PromotionError promotionError = new PromotionError(promotionSkuMap.get(skuInfoRequest.skuId).getSpuId(), skuInfoRequest.skuId, "不存在普通促销方案");
                submitResult.errorList.add(promotionError);
                logger.error(promotionError.toString());
            } else {
                PriceSchemeConfigRequest priceSchemeConfig = new PriceSchemeConfigRequest();
                priceSchemeConfig.weight = ActivityPriceSchemeConstants.weight;
                priceSchemeConfig.referenceType = ActivityPriceSchemeConstants.referenceType;
                priceSchemeConfig.referenceNo = ActivityPriceSchemeConstants.referenceNoPrefix + PromotionConstants.promotionId;
                priceSchemeConfig.skuId = skuInfoRequest.skuId;
                priceSchemeConfig.contractDetailId = skuGeneralPriceSchemeMap.get(skuInfoRequest.skuId).contractDetailId;
                priceSchemeConfig.startTime = activityPromotion.getStartTime();
                priceSchemeConfig.endTime = activityPromotion.getEndTime();
                priceSchemeConfig.price = promotionSkuMap.get(skuInfoRequest.skuId).getPromotionPrice();
                priceSchemeConfigList.add(priceSchemeConfig);
            }
        });
        return priceSchemeConfigList;
    }
}

2、外部调用

        BigDecimal totalSize = BigDecimal.valueOf(skuInfoRequestList.size());
        int threadCount = totalSize.divide(globalBatchSize, BigDecimal.ROUND_UP).intValue();

        List<Future<SubmitResult>> futureList = new ArrayList<>();
        for (int index = 0; index < threadCount; index++) {
            int start = index * globalBatchSize.intValue();
            int end = (index + 1) * globalBatchSize.intValue();
            if (end > skuInfoRequestList.size()) {
                end = skuInfoRequestList.size();
            }
            List<SkuInfoRequest> skuInfoRequestListForThread = skuInfoRequestList.subList(start, end);
            ExecuteSetPriceSchemeThread executeSetPriceSchemeThread = new ExecuteSetPriceSchemeThread(skuInfoRequestListForThread, promotionInfo, contractClient, innerPriceClient);
            futureList.add(ActivityThreadPool.executors.submit(executeSetPriceSchemeThread));
        }
        Set<Integer> errorSkuIdSet = new HashSet<>();
        for (Future<SubmitResult> future : futureList) {
            try {
                List<PromotionError> errorList = future.get().errorList;
                errorList.forEach(promotionError -> {
                    errorSkuIdSet.add(promotionError.getSkuId());
                });
                promotionBaseResponse.errorList.addAll(errorList);
            } catch (Exception e) {
                logger.error("", e);
            }
        }

转载于:https://my.oschina.net/u/3581789/blog/1559457