前言
很久没有写点东西了,主要是工作不忙,所忙的工作也都是比较简单的,并未遇到我没有接触过的技术,之前一直有一个需求就是对字段加密,而需要加密的字段有311万多,最初都是用单线程去写的,一次查询2000条,然后加密后一条一条的更新,结果一算,311万得花费5.7个小时,太慢了,于是就看了同事之前写的多线程(以前也写过,好久不写,必须copy现成的,哈哈哈哈),拷贝过来发现我的阿里编码规约提示:手动创建线程池,效果会更好哦
解决过程
参考博客
我之前通篇看过阿里编码规约,这块说到了我才有印象。网上查了一下,参考了这几篇博客
1、阿里不推荐使用Executors创建线程池
2、阿里为何不推荐使用Executors来创建线程池
3、阿里代码规约:手动创建线程池,效果会更好哦
4、阿里巴巴开发规范——不允许使用Executors来创建线程池
同学们可以先看看这几篇博客,而我站在了巨人的肩膀上,我就偷个懒,这里就不赘述啦,需要的自取,所以直接把我最后的成果贴出来,然后溜之大吉了。。。。。。
最终代码实现
public Integer ecryptCardNumberAll(){
//userDao已经注入过了,这里就直接用了
Integer sumLine = userDao.getSumLine(); //查询出要更新加密字段的总行数,我是一次查询2000条
//向上取整,比如手机计算器除出来的结果是1.05,
//这个RoundingMode.CEILING的结果就是2,
//也就是我们要的结果,2005条,每次查询2000天,剩下的5条我们肯定再查询一次。
//我上个月写过类似的文章,文章链接也一并献上:https://blog.csdn.net/fhf2424045058/article/details/108622932
Integer cycleTime = new BigDecimal(sumLine).divide(
new BigDecimal(2000), 0, RoundingMode.CEILING).intValue();
//不满200条的cycleTime为0,但为0的话,下面的for循环就进行不下去了,所以得加个1.
cycleTime = (cycleTime == 0) ? 1 : cycleTime;
log.info("共有:{} 条数据, 一次更新2000条,需要查询的次数为 :{}",sumLine,cycleTime);
Instant start = Instant.now();
final Integer[] flag = {0};
ThreadFactory namedThreadFactory = new ThreadFactoryBuilder()
.setNameFormat("thread-call-runner-%d").build();
ExecutorService taskExe = new ThreadPoolExecutor(4,6,200L,
TimeUnit.MILLISECONDS,new LinkedBlockingQueue<>(),
namedThreadFactory,new ThreadPoolExecutor.AbortPolicy());
//计数器
CountDownLatch latch = new CountDownLatch(cycleTime);
try {
for(int i = 0; i < cycleTime ;i++){
taskExe.submit(() -> {
try {
//这里写的是数据库查询的dao方法。
System.out.println(
Thread.currentThread().getName()
+"\t"+taskExe.toString()+ " 开始更新数据---");
flag[0]++;
} catch (Exception e) {
e.printStackTrace();
} finally {
latch.countDown();
}
});
}
latch.await();
System.out.println("初始化 - 完成, 耗时ms = " +
Duration.between(start, Instant.now()).toMillis());
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
//关闭线程池
taskExe.shutdown();
}
return flag[0];
}
这段代码我是先在单元测试没问题再放到了我对应的servce类里,里面的userDao.getSumLine()在单元测试里直接写死成你想要的值就好。
以上就是我的工作经历的一个记录,仅供参考!
----------------------------2020-11-04 更新---------------------------
由于要更新的数据有350多万,而上面的多线程还是很慢,按照上面的方式来处理也得需要6个多小时,后面公司另外一位大神让我改成嵌套的for循环内建多线程,改完后上周五上线后,350多万只需要2个多小时,大大的节省了时间,今天在单元测试也复现了一下,代码如下:
@Test
public void test6() {
Instant start = Instant.now();
Integer sumLine = 2010;
Integer cycleTime = new BigDecimal(sumLine).divide(new BigDecimal(2000), 0, RoundingMode.CEILING).intValue();
cycleTime = (cycleTime == 0) ? 1 : cycleTime;
System.out.println("每次查询2000条,一共需要查询 " + cycleTime + " 次");
try {
for (int i = 0; i < cycleTime; i++) {
System.out.println("外循环 " + i + "次");
CountDownLatch latch = new CountDownLatch(cycleTime);
final Integer[] flag = {0};
ThreadFactory namedThreadFactory = new ThreadFactoryBuilder().setNameFormat("thread-call-runner-%d").build();
ExecutorService taskExe = new ThreadPoolExecutor(4, 6, 200L,
TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>(),
namedThreadFactory, new ThreadPoolExecutor.AbortPolicy());
int time = i;
for (int j = 0; j < 2000; j++) {
System.out.println("内循环 ---第 " + (j) + "次");
try {
taskExe.submit(() -> {
try {
System.out.println("成功更新 apply_user");//这里是业务处理
flag[0]++;
} catch (Exception e) {
e.printStackTrace();
} finally {
latch.countDown();
}
});
} catch (Exception e) {
e.printStackTrace();
} finally {
taskExe.shutdown();//线程关闭在for循环内
}
}
latch.await();
System.out.println("初始化 - 完成, 耗时ms = " + Duration.between(start, Instant.now()).toMillis());
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
我们的处理逻辑如下:从数据库中查询出来有3560817条数据,每次查询2000条来更新,计算后需要循环查询的次数为1781次,然后再1781次里面再嵌套一次循环,每次更新2000条,在内循环内创建多线程,但是我执行了这个之后报错了
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@928763c rejected from java.util.concurrent.ThreadPoolExecutor@3b6eb2ec[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at com.jobtbao.PercentTest.test7(PercentTest.java:160)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
找了几篇文章都不是我想要的,看到了这篇文章后,找到了自己的问题,我在2000条的循环中,每循环一次就把多线程给关掉了, 修改后代码如下
@Test
public void test6() {
Instant start = Instant.now();
Integer sumLine = 2010;//这里就写的小一些,看一下执行成果,否则太大会跑很久。
Integer cycleTime = new BigDecimal(sumLine).divide(new BigDecimal(2000), 0, RoundingMode.CEILING).intValue();
cycleTime = (cycleTime == 0) ? 1 : cycleTime;
System.out.println("每次查询2000条,一共需要查询 " + cycleTime + " 次");
try {
for (int i = 0; i < cycleTime; i++) {
System.out.println("外循环 " + i + "次");
CountDownLatch latch = new CountDownLatch(cycleTime);
ThreadFactory namedThreadFactory = new ThreadFactoryBuilder().setNameFormat("thread-call-runner-%d").build();
ExecutorService taskExe = new ThreadPoolExecutor(4, 6, 200L,
TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>(),
namedThreadFactory, new ThreadPoolExecutor.AbortPolicy());
int time = i;
try {
for (int j = 0; j < 2000; j++) {
System.out.println("内循环 ---第 " + (j) + "次");
taskExe.submit(() -> {
try {
System.out.println("成功更新 apply_user");//这里是业务处理
} catch (Exception e) {
e.printStackTrace();
} finally {
latch.countDown();
}
});
}
} catch (Exception e) {
e.printStackTrace();
} finally {
taskExe.shutdown();//线程关闭在for循环外
}
latch.await();
System.out.println("初始化 - 完成, 耗时ms = " + Duration.between(start, Instant.now()).toMillis());
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
如此就正常了。
总结
多线程的知识远不是这么简单,需要学习的东西还很多,任重而道远,正如一句话,你知道的越多,你不知道的越多,多去学习才是王道。
与君共勉