将待插入数据,先拆分成不同的集合,各自再调用本类,以免因为数据过多造成卡顿。
同时,增加入库时间统计的功能:
/**
* @Description 拆分list后使用线程入库
* @Author lx
* @Date 2019/11/26 15:56
* @Param tableName 入库表名 columnName入库字段 list值集合
* @Return
* @Exception
*/
private void insertByCutList(String tableName, String columnName, List<String> list) {
if (list == null || list.size() == 0) {
log.info("入库" + tableName + "数据为空");
return;
}
//list每1000个拆分
int count = 1000;
int listNum = list.size() / count == 0 ? 1 : list.size() / count;
try {
List<String> newList = null;
long timeMillis = System.currentTimeMillis();
for (int i = 0; i < listNum; i++) {
//计算每个线程执行的数据
if ((i + 1) == listNum) {
int startIndex = (i * count);
int endIndex = list.size();
newList = list.subList(startIndex, endIndex);
} else {
int startIndex = (i * count);
int endIndex = (i + 1) * count;
newList = list.subList(startIndex, endIndex);
}
execThreadToInsert(tableName, columnName, newList);
}
long timeMillis2 = System.currentTimeMillis();
long times = timeMillis2 - timeMillis;
Calendar c = Calendar.getInstance();
c.setTimeInMillis(times);
log.info("=>" + tableName + "入库运行时长:" + c.get(Calendar.MINUTE) + "分钟" + c.get(Calendar.SECOND) + "秒" + c.get(Calendar.MILLISECOND) + "毫秒" + "<=");
} catch (Exception e) {
log.error("数据插入失败", e);
}
}
execThreadToInsert线程入库代码部分:
/**
* @Description 线程插入数据
* @Author lx
* @Date 2019/11/19 20:13
* @Param tableName 入库表名 columnName列名(多个字段名逗号分隔形式) list列对应值(值或对象的toString逗号分隔形式)
* @Return
* @Exception
*/
public void execThreadToInsert(String tableName, String columnName, List<String> list) throws Exception {
if (list == null || list.size() == 0) {
log.info("入库" + tableName + "时无数据");
return;
}
//数据集合大小
int listSize = list.size();
//开启的线程数
int runSize = 20;
//一个线程处理数据条数
int count = listSize / runSize;
if (count == 0) {
runSize = 1;
}
//存放每个线程的执行数据
List<String> newlist = null;
//创建一个线程池,数量和开启线程的数量一样
ExecutorService executor = Executors.newFixedThreadPool(runSize);
//创建两个个计数器
CountDownLatch begin = new CountDownLatch(1);
CountDownLatch end = new CountDownLatch(runSize);
//循环创建线程
for (int i = 0; i < runSize; i++) {
//计算每个线程执行的数据
if ((i + 1) == runSize) {
int startIndex = (i * count);
int endIndex = list.size();
newlist = list.subList(startIndex, endIndex);
} else {
int startIndex = (i * count);
int endIndex = (i + 1) * count;
newlist = list.subList(startIndex, endIndex);
}
List<String> finalNewlist = newlist;
executor.execute(new Runnable() {
@Override
public void run() {
try {
//执行插入
XXXXMapper.insertToDB(tableName, columnName, finalNewlist);
//执行完让线程直接进入等待
begin.await();
} catch (Exception e) {
log.error("线程插入失败,tableName:" + tableName, e);
} finally {
//当一个线程执行完 了计数要减一不然这个线程会被一直挂起
//end.countDown(),这个方法就是直接把计数器减一的
end.countDown();
}
}
});
}
begin.countDown();
end.await();
//执行完关闭线程池
executor.shutdown();
}
执行插入部分代码略,项目使用的是mybatis,调用对应的映射接口。