多线程批量导入

多线程批量导入

一、切入主题

本文主要解决：

 

1. 多线程Excel导入
2. 解决分布式事务

 

二、需求说明

分布式系统，向系统中批量导入信息，信息需要存储在新生表、学生表、权限表。

步骤：

1.得到三个list ，分别批量向数据库中存。

2.使用多线程对分批进入数据库

情况说明：

         因为是学生和新生是一个库，用户是一个表。如果两个库在不同的物理节点上，就会有分布式事务的问题。例如向学生和新生插入成功，用户表插入失败，学生和新生表不会回滚。

 

三、问题再现

 

@Transactional(rollbackFor = Exception.class)     public int addFreshStudentsNew(List<FreshStudentAndStudentModel> list, String schoolNo) {         if (list == null || list.isEmpty()) {             return 0;         }         List<StudentEntity> studentEntityList = new LinkedList<>();         List<EnrollStudentEntity> enrollStudentEntityList = new LinkedList<>();         List<AllusersEntity> allusersEntityList = new LinkedList<>();           list.forEach(freshStudentAndStudentModel -> {             EnrollStudentEntity enrollStudentEntity = new EnrollStudentEntity();             StudentEntity studentEntity = new StudentEntity();             BeanUtils.copyProperties(freshStudentAndStudentModel, studentEntity);             BeanUtils.copyProperties(freshStudentAndStudentModel, enrollStudentEntity);             String operator = TenancyContext.UserID.get();             String studentId = BaseUuidUtils.base58Uuid();             enrollStudentEntity.setId(BaseUuidUtils.base58Uuid());             enrollStudentEntity.setStudentId(studentId);             enrollStudentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());             enrollStudentEntity.setOperator(operator);             studentEntity.setId(studentId);             studentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());             studentEntity.setOperator(operator);             studentEntityList.add(studentEntity);             enrollStudentEntityList.add(enrollStudentEntity);               AllusersEntity allusersEntity = new AllusersEntity();             allusersEntity.setId(enrollStudentEntity.getId());             allusersEntity.setUserCode(enrollStudentEntity.getNemtCode());             allusersEntity.setUserName(enrollStudentEntity.getName());             allusersEntity.setSchoolNo(schoolNo);             allusersEntity.setTelNum(enrollStudentEntity.getTelNum());             allusersEntity.setPassword(enrollStudentEntity.getNemtCode());  //密码设置为考生号             allusersEntityList.add(allusersEntity);         });         //开启50个线程         int nThreads = 50;           int size = enrollStudentEntityList.size(); //自定义线程池         ExecutorService executorService = Executors.newFixedThreadPool(nThreads);         List<Future<Integer>> futures = new ArrayList<Future<Integer>>(nThreads);         for (int i = 0; i < nThreads; i++) {               final List<EnrollStudentEntity> EnrollStudentEntityImputList = enrollStudentEntityList.subList(size / nThreads * i, size / nThreads * (i + 1));             final List<StudentEntity> studentEntityImportList = studentEntityList.subList(size / nThreads * i, size / nThreads * (i + 1));             final List<AllusersEntity> allusersEntityImportList = allusersEntityList.subList(size / nThreads * i, size / nThreads * (i + 1));              Callable<Integer> task1 = () -> {         enrollStudentDao.insertAll(EnrollStudentEntityImputList);         studentDao.insertAll(studentEntityImportList);         allusersFacade.insertUserList(allusersEntityImportList);       [王雷1]      };             futures.add(executorService.submit(task1));         }           executorService.shutdown();           if (!futures.isEmpty() && futures != null) {             return 10;         }         return -10;     }

 

以上代码说明：

 

         1.很清楚的可以看出是serviceImpl的代码，其中使用了多线程。使用自定义线程池，设置了50个线程，每个线程对list进行分组，分组后依次插入到数据库中。

 

2.另外每条线程有三次数据库插入操作，前两个插入是基础数据库，后面的是权限数据库。当向权限表插入数据的时候，前两张表的数据没有回滚。

 

明明整个方法前面已经加上@Transactional注解，为什么会发生不会滚的问题呢？

 

失败原因说明：

 

Service类内部方法调用

 

举个简单的例子：

 

@Service public class RabbitServiceImpl implements RabbitService {       @Autowired     private RabbitDao rabbitDao;     @Autowired     private TortoiseDao tortoiseDao;         @Override     public Rabbit methodA(String name){         return methodB(name);     }         @Transactional(propagation = Propagation.REQUIRED, isolation = Isolation.READ_COMMITTED)     public boolean methodB(String name){         rabbitDao.insertRabbit(name);         tortoiseDao.insertTortoise(name);         return true;     }     }

 

 

 

在方法 A 中调用方法 B，实际上是通过“this”的引用，也就是直接调用了目标类的方法，而非通过 Spring 上下文获得的代理类，所以事务是不会开启的。

 

         在本例中addFreshStudentsNew方法中，三个insert就等价于一个方法。是通过this调用的，没有开启事务。

四、解决思路

把三个insert方法抽取出来放到一个service中，这个service可以被spring管理。

 

@Transactional(rollbackFor = Exception.class)     public int addFreshStudentsNew(List<FreshStudentAndStudentModel> list, String schoolNo) {         if (list == null || list.isEmpty()) {             return 0;         }         List<StudentEntity> studentEntityList = new LinkedList<>();         List<EnrollStudentEntity> enrollStudentEntityList = new LinkedList<>();         List<AllusersEntity> allusersEntityList = new LinkedList<>();           list.forEach(freshStudentAndStudentModel -> {             EnrollStudentEntity enrollStudentEntity = new EnrollStudentEntity();             StudentEntity studentEntity = new StudentEntity();             BeanUtils.copyProperties(freshStudentAndStudentModel, studentEntity);             BeanUtils.copyProperties(freshStudentAndStudentModel, enrollStudentEntity);             String operator = TenancyContext.UserID.get();             String studentId = BaseUuidUtils.base58Uuid();             enrollStudentEntity.setId(BaseUuidUtils.base58Uuid());             enrollStudentEntity.setStudentId(studentId);             enrollStudentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());             enrollStudentEntity.setOperator(operator);             studentEntity.setId(studentId);             studentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());             studentEntity.setOperator(operator);             studentEntityList.add(studentEntity);             enrollStudentEntityList.add(enrollStudentEntity);               AllusersEntity allusersEntity = new AllusersEntity();             allusersEntity.setId(enrollStudentEntity.getId());             allusersEntity.setUserCode(enrollStudentEntity.getNemtCode());             allusersEntity.setUserName(enrollStudentEntity.getName());             allusersEntity.setSchoolNo(schoolNo);             allusersEntity.setTelNum(enrollStudentEntity.getTelNum());             allusersEntity.setPassword(enrollStudentEntity.getNemtCode());  //密码设置为考生号             allusersEntityList.add(allusersEntity);         });             int nThreads = 50;           int size = enrollStudentEntityList.size();         ExecutorService executorService = Executors.newFixedThreadPool(nThreads);         List<Future<Integer>> futures = new ArrayList<Future<Integer>>(nThreads);             for (int i = 0; i < nThreads; i++) {               final List<EnrollStudentEntity> EnrollStudentEntityImputList = enrollStudentEntityList.subList(size / nThreads * i, size / nThreads * (i + 1));             final List<StudentEntity> studentEntityImportList = studentEntityList.subList(size / nThreads * i, size / nThreads * (i + 1));             final List<AllusersEntity> allusersEntityImportList = allusersEntityList.subList(size / nThreads * i, size / nThreads * (i + 1));              Callable<Integer> task1 = () -> {             studentSave.saveStudent(EnrollStudentEntityImputList,studentEntityImportList,allusersEntityImportList);                return 1;             };           futures.add(executorService.submit(task1));         }           executorService.shutdown();           if (!futures.isEmpty() && futures != null) {             return 10;         }         return -10;     }

 

新建立一个StudentSave   的service：

 

package com.dmsdbj.itoo.basicInfo.service;   import com.dmsdbj.itoo.authorizationManagement.entity.AllusersEntity; import com.dmsdbj.itoo.basicInfo.entity.EnrollStudentEntity; import com.dmsdbj.itoo.basicInfo.entity.StudentEntity;   import java.util.List;   /**  * Created by Ares on 2018/3/26.  */ public interface StudentSave {      void saveStudent(final List<EnrollStudentEntity> EnrollStudentEntityImputList , final List<StudentEntity> studentEntityImportList , final List<AllusersEntity> allusersEntityImportList );       }

 

Service的实现类：

package com.dmsdbj.itoo.basicInfo.service.impl;   import com.alibaba.dubbo.config.annotation.Reference; import com.dmsdbj.itoo.authorizationManagement.entity.AllusersEntity; import com.dmsdbj.itoo.authorizationManagement.facade.AllusersFacade; import com.dmsdbj.itoo.basicInfo.dao.EnrollStudentDao; import com.dmsdbj.itoo.basicInfo.dao.StudentDao; import com.dmsdbj.itoo.basicInfo.entity.EnrollStudentEntity; import com.dmsdbj.itoo.basicInfo.entity.StudentEntity; import com.dmsdbj.itoo.basicInfo.service.*; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; import org.springframework.transaction.annotation.Transactional;   import java.util.List;   /**  * Created by Ares on 2018/3/26.  */ @Component("studentSave") public class StudentSaveImpl implements StudentSave {       @Reference(url = "dubbo://192.168.21.95:20870")     private AllusersFacade allusersFacade;       @Autowired     private StudentDao studentDao;     @Autowired     private EnrollStudentDao enrollStudentDao;         @Transactional(rollbackFor = Exception.class )     @Override     public void saveStudent(final List<EnrollStudentEntity> EnrollStudentEntityImputList , final List<StudentEntity> studentEntityImportList , final List<AllusersEntity> allusersEntityImportList ){         enrollStudentDao.insertAll(EnrollStudentEntityImputList);         studentDao.insertAll(studentEntityImportList);         allusersFacade.insertUserList(allusersEntityImportList);     } }

 

 

亲测：当调用的三个插入数据库在同一个物理节点上，事务可以成功。当调用的三个插入的数据库在不同的物理节点上，事务依然可以生效。

 

 

五、更多指导

 

http://rabbitgyk.com/2018/01/27/why-not-rollback-spring-transaction/

 

---

 [王雷1]这里是主要对数据库操作的地方