java中多个并行输入一个输出,java – Reader#lines()由于其拼接器中的不可配置的批量大小策略而非常并行化...

这里是答案,在Spliterators.IteratorSpliterator的源代码中,BufferedReader#lines()使用的代码：

@Override

public Spliterator trySplit() {

/*

* Split into arrays of arithmetically increasing batch

* sizes. This will only improve parallel performance if

* per-element Consumer actions are more costly than

* transferring them into an array. The use of an

* arithmetic progression in split sizes provides overhead

* vs parallelism bounds that do not particularly favor or

* penalize cases of lightweight vs heavyweight element

* operations,across combinations of #elements vs #cores,* whether or not either are known. We generate

* O(sqrt(#elements)) splits,allowing O(sqrt(#cores))

* potential speedup.

*/

Iterator extends T> i;

long s;

if ((i = it) == null) {

i = it = collection.iterator();

s = est = (long) collection.size();

}

else

s = est;

if (s > 1 && i.hasNext()) {

int n = batch + BATCH_UNIT;

if (n > s)

n = (int) s;

if (n > MAX_BATCH)

n = MAX_BATCH;

Object[] a = new Object[n];

int j = 0;

do { a[j] = i.next(); } while (++j < n && i.hasNext());

batch = j;

if (est != Long.MAX_VALUE)

est -= j;

return new ArraySpliterator<>(a,j,characteristics);

}

return null;

}

也值得注意的是常数：

static final int BATCH_UNIT = 1 << 10; // batch array size increment

static final int MAX_BATCH = 1 << 25; // max batch array size;

所以在我的例子中,我使用6,000个元素,因为批量大小为1024,所以我只需要批量批次.这正好解释了我的观察结果：最初使用三个内核,当两个小批次完成时,它们都会被丢弃.在此期间,我尝试了一个具有6万个元素的修改示例,然后我得到几乎100％的cpu利用率.

为了解决我的问题,我已经开发了下面的代码,它允许我将任何现有流转换成一个Spliterator#trySplit将其分割成指定大小的批次.从我的问题使用它的最简单的方法是这样的：

toFixedBatchStream(Files.newBufferedReader(inputPath).lines(),20)

在较低级别上,下面的类是一个Spliterator包装器,它改变了包装的spliterator的trySplit行为,并保留其他方面不变.

import static java.util.Spliterators.spliterator;

import static java.util.stream.StreamSupport.stream;

import java.util.Comparator;

import java.util.Spliterator;

import java.util.function.Consumer;

import java.util.stream.Stream;

public class FixedBatchSpliteratorWrapper implements Spliterator {

private final Spliterator spliterator;

private final int batchSize;

private final int characteristics;

private long est;

public FixedBatchSpliteratorWrapper(Spliterator toWrap,long est,int batchSize) {

final int c = toWrap.characteristics();

this.characteristics = (c & SIZED) != 0 ? c | SUBSIZED : c;

this.spliterator = toWrap;

this.est = est;

this.batchSize = batchSize;

}

public FixedBatchSpliteratorWrapper(Spliterator toWrap,int batchSize) {

this(toWrap,toWrap.estimateSize(),batchSize);

}

public static Stream toFixedBatchStream(Stream in,int batchSize) {

return stream(new FixedBatchSpliteratorWrapper<>(in.spliterator(),batchSize),true);

}

@Override public Spliterator trySplit() {

final HoldingConsumer holder = new HoldingConsumer<>();

if (!spliterator.tryAdvance(holder)) return null;

final Object[] a = new Object[batchSize];

int j = 0;

do a[j] = holder.value; while (++j < batchSize && tryAdvance(holder));

if (est != Long.MAX_VALUE) est -= j;

return spliterator(a,characteristics());

}

@Override public boolean tryAdvance(Consumer super T> action) {

return spliterator.tryAdvance(action);

}

@Override public void forEachRemaining(Consumer super T> action) {

spliterator.forEachRemaining(action);

}

@Override public Comparator super T> getComparator() {

if (hasCharacteristics(SORTED)) return null;

throw new IllegalStateException();

}

@Override public long estimateSize() { return est; }

@Override public int characteristics() { return characteristics; }

static final class HoldingConsumer implements Consumer {

Object value;

@Override public void accept(T value) { this.value = value; }

}

}