如何从一亿个数内得到最大的100个数，比较方案的时间消耗

实验材料：

• 文件（内部为1亿个数字每个数字一行），如图
• 系统如图：
• 计算时间接口：

public interface Runtime {
    default void RunT() throws Exception {
        Instant start = Instant.now();
        this.run();
        Instant end = Instant.now();
        System.out.println("\n"+"**** process useTime:" + Duration.between(start, end).toMillis() + "ms");
    };
    void run() throws Exception;

}

• 计算方法

public class CreatFile implements Runtime{
    private static final int Threshould=100000000;
    public static void main(String[] args) throws Exception {
      new CreatFile().RunT();
    }
    @Override
    public void run() throws IOException {
            findMax0("rawData0.txt",100);
    }
    public void findMax0(String path,int MaxNum) throws IOException {
        BufferedReader bufferedReader = new BufferedReader(new FileReader(new File(path)));
        int[] maxNum = new int[100];
        String s="";int i=0;
        while ((s!=null)){
            s = bufferedReader.readLine();
            if (s==null){
                break;
            }
            Integer integer = Integer.valueOf(s);
            insert(maxNum,integer);
            i++;
        }
        System.out.println("maxNum:");
        for (int a:maxNum){
            System.out.print(a+" - ");
        }
    }
    void insert(int[]arr,int a){
        for(int i=0;i<arr.length;i++){
            if (a<arr[i]){
                break;
            }
            else {
                if(i==0){
                    arr[0]=a;
                }
                else {
                    arr[i-1]=arr[i];
                    arr[i]=a;
                }
            }
        }
    }
    public static void write() throws IOException {
        FileWriter fileWriter = new FileWriter(new File("rawData0.txt"), false);
        int i=0;
        while (i<Threshould){
                fileWriter.write(i+"");
            fileWriter.write(System.lineSeparator());
            i++;
        }
        fileWriter.close();
    }
    public static void read() throws IOException {
        BufferedReader bufferedReader = new BufferedReader(new FileReader(new File("rawData0.txt")));
        byte[] bytes = new byte[1024];
        String s="";
        int i=0;
        while ((s!=null)&&i<100){
             s = bufferedReader.readLine();
             System.out.println((i++)+" line:  "+s.length()+"--->"+s);
             System.out.println("##################################");
        }

    }

}

SPSC：单生产者–对--单消费者

 public void findMax0(String path,int MaxNum) throws IOException {
        BufferedReader bufferedReader = new BufferedReader(new FileReader(new File(path)));
        int[] maxNum = new int[100];
        String s="";int i=0;
        while ((s!=null)){
            s = bufferedReader.readLine();
            if (s==null){
                break;
            }
            Integer integer = Integer.valueOf(s);
            insert(maxNum,integer);
            i++;
        }
        System.out.println("maxNum:");
        for (int a:maxNum){
            System.out.print(a+" - ");
        }
    }

• 读取数据，比较，替换最大的100个最后得到最大得100个数
  

SPMC：单生产者-多消费者

 private static final int Threshould=100000000;
    private static final int MaxNum=100;
    private static final int RoundCount= (int) Math.sqrt((double) Threshould*MaxNum);

• 这里1亿个数，100个最大值，取乘积的平方根这样分为1000个小任务从10万个数获取100个最大值，一共有1000*100=10万个值，刚好再做一次小任务，简化了代码实现，节约了大概一半时间。

    public void findMaxBySPMC(String path,int MaxNum) throws IOException, InterruptedException {
        BufferedReader bufferedReader = new BufferedReader(new FileReader(new File(path)));
        String s="";int i=0;
        int[] last=new int[RoundCount];
        int[] MaxAll=new int[MaxNum];
        AtomicInteger Position = new AtomicInteger(0);
        ExecutorService executorService = Executors.newFixedThreadPool(java.lang.Runtime.getRuntime().availableProcessors());
        class findMaxTask implements Runnable{
            private int[]arr;
            private int[] maxNum;
            public findMaxTask(int[] arr,int[]maxNum){
                this.arr=arr;
                this.maxNum=maxNum;
            }
            @Override
            public void run() {
                for (int i = 0;i<arr.length;i++) {
                    insert(maxNum,arr[i]);
                }
                //双重判断，最后一个任务结束
                if (Position.get()>=RoundCount){
                    System.out.println(Thread.currentThread().getName()+"MaxAll[]:");
                    for (int a:MaxAll){
                        System.out.print(a+" - ");
                    }
                    System.out.println("");
                    executorService.shutdown();
                    return;
                }
                for(int i=0;i<MaxNum;i++){
                    last[Position.getAndIncrement()]=maxNum[i];
                }
                // 最后一个任务开始
                if (Position.get()>=RoundCount){
                    System.out.println("\n"+"last task-->"+Position.get());
                    findMaxTask lastTask = new findMaxTask(last, MaxAll);
                    executorService.submit(lastTask);
                }

            }
        }
        int[] ints=new int[RoundCount];int rounds=0;
        // while ((s!=null)&&rounds<2){
        while ((s!=null)){
            s = bufferedReader.readLine();
            //最后一次可能不满数组
            if (s==null){
                System.out.println("file end ###"+" []Length : "+i);
                if (i!=0) {
                    findMaxTask findMaxTask = new findMaxTask(ints,new int[MaxNum]);
                    rounds++;
                    executorService.submit(findMaxTask);
                }
                break;
            }
            Integer integer = Integer.valueOf(s);
            ints[i]=integer;
            i++;
            if (i>=ints.length) {
                findMaxTask findMaxTask = new findMaxTask(ints,new int[MaxNum]);
                executorService.submit(findMaxTask);
                i=0;
                int[] ints0=new int[RoundCount];
                ints=ints0;
                rounds++;
            }
        }
        while (!executorService.awaitTermination(10, TimeUnit.MILLISECONDS)) {
        }
    }

这也是map-reduce的最初原型

• 分成多个小任务就是map阶段
• 小任务输出汇总到一起，last任务就是reduce阶段
  参考信息

缺点：只有一个读线程，任务线程基本在等待读线程。

从线程的运行看得出只有pool-1-thread-3（last任务），pool-1-thread-2（小任务）执行了，其他线程都是闲置。