多线程将数据源分段取值

最新推荐文章于 2024-06-13 21:00:00 发布

悠_ran

最新推荐文章于 2024-06-13 21:00:00 发布

阅读量309

点赞数

分类专栏：日常笔记

本文链接：https://blog.csdn.net/you_ran_jian/article/details/103089031

版权

日常笔记专栏收录该内容

20 篇文章 0 订阅

订阅专栏

今天在做一个大量数据的数据库操作,需要把数据插入到数据库中,但是单线程查实在太慢即使批量查询还是慢,就想到了多线程.
上代码:
list是我取出来的数据源

  int thread = 200;// 线程数
             if (list.size()<20) {
                 thread=1;
             }
             int num = list.size();//一共有多少数据
             final int avg = num / thread;// 每个线程执行的个数
             List<Thread> listThreads = new ArrayList<Thread>();
             for (int k = 0; k < thread; k++) {
                 // 20个线程一起工作
                 final int count = k;
                 listThreads.add(new Thread("我是"+(p+k+1)+"线程") {
                     @Override
                     public void run() {
                         //多线程运行方法体 下面是方法体
                         List<HistoryData> listH = new ArrayList<>();
                         for (int j = 0; j < avg; j++) {
                             int number = j + count * avg;//当前执行的数据
                             JSONObject jsonObjectRequest = new JSONObject();
                             JSONObject jsonObjectReponse = new JSONObject();
                             JSONArray jsonArrayRequest = new JSONArray();
                             JSONArray jsonArrayReponse = new JSONArray();

                             String oldname = list.get(number).get_id();
                             try {
                                 jsonObjectRequest = JSON.parseObject(list.get(number).getRequest());
                                 jsonObjectReponse = JSON.parseObject(list.get(number).getReponse());
                                 jsonArrayRequest = jsonObjectRequest.getJSONArray("cars");
                                 jsonArrayReponse = jsonObjectReponse.getJSONArray("result");
                             } catch (Exception e) {
                                 continue;
                             }
                             if (jsonArrayRequest != null && jsonArrayReponse != null) {
                                 for (int o = 0; o< jsonArrayRequest.size(); o++) {
                                     String platNumber = jsonArrayRequest.getJSONObject(o   ).getString("platNumber");
                                     String engineNumber = jsonArrayRequest.getJSONObject(o).getString("engineNumber");
                                     historyData.setPlatNumber(platNumber);
                                     historyData.setEngineNumber(engineNumber);
                                     historyData.setData(jsonArrayReponse.getJSONObject(o).toString());
                                     historyData.setData1(jsonArrayRequest.getJSONObject(o).toString());
                                     historyData.setOldid(oldname);
                                     historyData.setSum(String.valueOf( jsonArrayRequest.size()));
                                     historyData.setNumber(String.valueOf(number));
                                     historyData.setThreaNmae(Thread.currentThread().getName());
                                     listH.add(historyData);
                      
                                 }
                             }
                         }
                         //mongodb批量插入
                         mongoTemplate.insert(listH,HistoryData.class);

                     }
                });

             }
             for (int k = 0; k < thread; k++) {//运行多线程
                 listThreads.get(k).start();
             }
             try {
                 for (int k = 0; k < thread; k++) {//等等多线程运行完成
                     listThreads.get(k).join();
                 }

             } catch (InterruptedException e) {
                 e.printStackTrace();
             }

这样我们就不用担心锁的问题了,不过有一个问题:如果list数据个数除以线程数还有剩余那么这样我们的数据就丢了,所以我们还要在下面单独处理这些数据
单线程处理剩下的数据也不会超过我们的线程数所以单线程操作也是很快的

 int ynum = num-(avg*thread);//剩余的数据
             for(int t=0;t<ynum;t++) {
                 JSONObject jsonObjectRequest = new JSONObject();
                 JSONObject jsonObjectReponse = new JSONObject();
                 JSONArray jsonArrayRequest = new JSONArray();
                 JSONArray jsonArrayReponse = new JSONArray();
                 try {
                     jsonObjectRequest = JSON.parseObject(list.get((avg*thread)+t).getRequest());
                     jsonObjectReponse = JSON.parseObject(list.get((avg*thread)+t).getReponse());
                     jsonArrayRequest = jsonObjectRequest.getJSONArray("cars");
                     jsonArrayReponse = jsonObjectReponse.getJSONArray("result");
                 } catch (Exception e) {
                     continue;
                 }
                 if (jsonArrayRequest != null && jsonArrayReponse != null) {
                     for (int o = 0; o< jsonArrayRequest.size(); o++) {
                         String platNumber = jsonArrayRequest.getJSONObject(o).getString("platNumber");
                         String engineNumber = jsonArrayRequest.getJSONObject(o).getString("engineNumber");
                         historyData.setPlatNumber(platNumber);
                         historyData.setEngineNumber(engineNumber);
                         historyData.setData(jsonArrayReponse.getJSONObject(o).toString());
                         mongoTemplate.insert(historyData, "history_data");
                     }
                 }
             }