这里最重要的问题是你的情况是什么瓶颈。
如果瓶颈是您的磁盘IO,那么在软件部分可以做的不多。并行计算只会使事情变得更糟,因为同时读取不同部分的文件会降低磁盘性能。
如果瓶颈是处理能力,并且有多个CPU内核,那么您可以利用启动多个线程来处理文件的不同部分。您可以安全地创建多个InputStreams或Readers来并行读取文件的不同部分(只要不超过操作系统对打开文件数量的限制)。您可以将工作分成任务并并行运行,如下例所示:
import java.io.*;
import java.util.*;
import java.util.concurrent.*;
public class Split {
private File file;
public Split(File file) {
this.file = file;
}
// Processes the given portion of the file.
// Called simultaneously from several threads.
// Use your custom return type as needed, I used String just to give an example.
public String processPart(long start, long end)
throws Exception
{
InputStream is = new FileInputStream(file);
is.skip(start);
// do a computation using the input stream,
// checking that we don't read more than (end-start) bytes
System.out.println("Computing the part from " + start + " to " + end);
Thread.sleep(1000);
System.out.println("Finished the part from " + start + " to " + end);
is.close();
return "Some result";
}
// Creates a task that will process the given portion of the file,
// when executed.
public Callable processPartTask(final long start, final long end) {
return new Callable() {
public String call()
throws Exception
{
return processPart(start, end);
}
};
}
// Splits the computation into chunks of the given size,
// creates appropriate tasks and runs them using a
// given number of threads.
public void processAll(int noOfThreads, int chunkSize)
throws Exception
{
int count = (int)((file.length() + chunkSize - 1) / chunkSize);
java.util.List> tasks = new ArrayList>(count);
for(int i = 0; i < count; i++)
tasks.add(processPartTask(i * chunkSize, Math.min(file.length(), (i+1) * chunkSize)));
ExecutorService es = Executors.newFixedThreadPool(noOfThreads);
java.util.List> results = es.invokeAll(tasks);
es.shutdown();
// use the results for something
for(Future result : results)
System.out.println(result.get());
}
public static void main(String argv[])
throws Exception
{
Split s = new Split(new File(argv[0]));
s.processAll(8, 1000);
}
}