我的业务是,把表里的6千万数据做某些处理,但是由于数据量太大,不可能一次性查出,只能分批处理。
我们打算每5W每5W取出数据进行处理,这批5W处理完自动进入下一批这种,所以需要监听到所有子线程结束后做好判断走入下一步
利用Future来监测@Async的回调,然后主线程进入while循环,每几秒判断一下Future的isDone状态,如果线程结束会返回true
我下面代码将5W数据进行平均拆分,拆分成5个list,分别进入5个线程进行处理,当5个线程全是true之后就可以判断子线程全执行完了
@RequestMapping(value = "/manager/filterData", method = RequestMethod.POST)
public void filterData() {
try {
Future<Integer> a = new AsyncResult<Integer>(1);
Future<Integer> b = new AsyncResult<Integer>(1);
Future<Integer> c = new AsyncResult<Integer>(1);
Future<Integer> d = new AsyncResult<Integer>(1);
Future<Integer> e = new AsyncResult<Integer>(1);
// 查询企查查的数据
List<QccBidInfo> qccBidInfoList = qccBidInfoService.selectListByCreateTime();
if (qccBidInfoList.size() > 0) {
List<List<QccBidInfo>> qccList = ListHelper.averageAssign(qccBidInfoList, 5);
a = iAsyncService.asyncQccBidInfoData(qccList.get(0));
b = iAsyncService.asyncQccBidInfoData(qccList.get(1));
c = iAsyncService.asyncQccBidInfoData(qccList.get(2));
d = iAsyncService.asyncQccBidInfoData(qccList.get(3));
e = iAsyncService.asyncQccBidInfoData(qccList.get(4));
}
while (true) {
System.out.println(a.isDone() + "-" + b.isDone() + "-" + c.isDone() + "-" + d.isDone() + "-" + e.isDone());
if (a.isDone() && b.isDone() && c.isDone() && d.isDone() && e.isDone()) {
System.out.println("结束处理================");
break;
}
Thread.sleep(5000);
}
} catch (Exception e) {
LoggerUtil.error(e.getMessage());
}
}
额外发下拆分list的,这个网上都是一样的写法
/**
* 将一个list均分成n个list,主要通过偏移量来实现的
*
* @param source
* @return
*/
public static <T> List<List<T>> averageAssign(List<T> source, int n) {
List<List<T>> result = new ArrayList<List<T>>();
int remaider = source.size() % n; //(先计算出余数)
int number = source.size() / n; //然后是商
int offset = 0;//偏移量
for (int i = 0; i < n; i++) {
List<T> value = null;
if (remaider > 0) {
value = source.subList(i * number + offset, (i + 1) * number + offset + 1);
remaider--;
offset++;
} else {
value = source.subList(i * number + offset, (i + 1) * number + offset);
}
result.add(value);
}
return result;
}