DataSketch求流式数据分位数

类似的功能Spark、Flink等都实现了，但是使用起来比较繁琐。Apache下的DataSketch用起来稍微简单一些。先导包：

<dependency>
     <groupId>org.apache.datasketches</groupId>
     <artifactId>datasketches-java</artifactId>
     <version>3.3.0</version>
</dependency>
<dependency>
     <groupId>org.apache.lucene</groupId>
     <artifactId>lucene-core</artifactId>
     <version>4.0.0</version>
</dependency>

用数组简单模拟流式数据：

public class DemoApplication {

    public static void main(String[] args) {
        UpdateDoublesSketch sketch = DoublesSketch.builder().build();
        int length = 10000000;
        double[] nums = new double[length];
        Random random = new Random();
        for (int i = 0; i < length; ++i) {
            nums[i] = random.nextDouble() * random.nextInt();
        }
        long start = System.currentTimeMillis();
        for (int i = 0; i < length; ++i) {
            sketch.update(nums[i]);
        }
        long insertCost = System.currentTimeMillis() - start;

        start = System.currentTimeMillis();
        Arrays.sort(nums);
        long sortCost = System.currentTimeMillis() - start;

        BigDecimal correct = new BigDecimal(Double.toString(nums[(int)(length * 0.95) - 1]));
        start = System.currentTimeMillis();
        BigDecimal approximate = new BigDecimal(Double.toString(sketch.getQuantile(0.95)));
        long sketchCost = System.currentTimeMillis() - start + insertCost;
        System.out.println("original array memory usage:" + RamUsageEstimator.humanSizeOf(nums));
        System.out.println("sort time:" + sortCost + "ms");
        System.out.println("data sketch memory usage: " + RamUsageEstimator.humanSizeOf(sketch));
        System.out.println("sketch insert time:" + insertCost + "ms");
        System.out.println("sketch total time:" + sketchCost + "ms");
        System.out.println("correct:" + correct);
        System.out.println("sketch:" + approximate);
        System.out.println("error percentage:" + approximate.subtract(correct).divide(correct, 6, RoundingMode.HALF_UP).abs());
    }
}

1w数据：

10w数据：

100w数据：

1000w数据

5000w数据

自己试下来，误差在1%~5%波动，属于可以接受的范围，而且DataSketch的使用内存非常非常少，不会随数据增长线性增加。最主要是数据不需要常驻内存，而且不用每次都排序，在海量数据求分位数时比较实用。