java多线程按行读取文件_java – 多线程读取大量文件

你可能偶然选择了并行活动的绝对最糟糕的例子！

从单个机械磁盘并行读取实际上比使用单个线程读取要慢,因为实际上,当每个线程轮到运行时,您将机械磁头弹回到磁盘的不同部分.这最好保留为单线程活动.

让我们再举一个例子,它与你的类似,但实际上可以提供一些好处：假设我想在一个巨大的单词列表中搜索某个单词的出现(这个列表甚至可能来自一个磁盘文件,但是像我一样说,由一个线程读取).假设我可以像你的例子一样使用3个线程,每个线程搜索巨大单词列表的1/3,并保持一个本地计数器显示搜索单词出现的次数.

在这种情况下,您需要将列表分为3个部分,将每个部分传递给其类型实现Runnable的不同对象,并在run方法中实现搜索.

运行时本身不知道如何进行分区或类似的东西,你必须自己指定.还有许多其他分区策略,每个策略都有自己的优点和缺点,但我们现在可以坚持使用静态分区.

我们来看一些代码：

class SearchTask implements Runnable {

private int localCounter = 0;

private int start; // start index of search

private int end;

private List words;

private String token;

public SearchTask(int start, int end, List words, String token) {

this.start = start;

this.end = end;

this.words = words;

this.token = token;

}

public void run() {

for(int i = start; i < end; i++) {

if(words.get(i).equals(token)) localCounter++;

}

}

public int getCounter() { return localCounter; }

}

// meanwhile in main :)

List words = new ArrayList();

// populate words

// let's assume you have 30000 words

// create tasks

SearchTask task1 = new SearchTask(0, 10000, words, "John");

SearchTask task2 = new SearchTask(10000, 20000, words, "John");

SearchTask task3 = new SearchTask(20000, 30000, words, "John");

// create threads for each task

Thread t1 = new Thread(task1);

Thread t2 = new Thread(task2);

Thread t3 = new Thread(task3);

// start threads

t1.start();

t2.start();

t3.start();

// wait for threads to finish

t1.join();

t2.join();

t3.join();

// collect results

int counter = 0;

counter += task1.getCounter();

counter += task2.getCounter();

counter += task3.getCounter();

这应该很好.请注意,在实际情况下,您将构建更通用的分区方案.如果您希望返回结果,也可以使用ExecutorService并实现Callable而不是Runnable.

所以使用更高级的结构的另一个例子：

class SearchTask implements Callable {

private int localCounter = 0;

private int start; // start index of search

private int end;

private List words;

private String token;

public SearchTask(int start, int end, List words, String token) {

this.start = start;

this.end = end;

this.words = words;

this.token = token;

}

public Integer call() {

for(int i = start; i < end; i++) {

if(words.get(i).equals(token)) localCounter++;

}

return localCounter;

}

}

// meanwhile in main :)

List words = new ArrayList();

// populate words

// let's assume you have 30000 words

// create tasks

List tasks = new ArrayList();

tasks.add(new SearchTask(0, 10000, words, "John"));

tasks.add(new SearchTask(10000, 20000, words, "John"));

tasks.add(new SearchTask(20000, 30000, words, "John"));

// create thread pool and start tasks

ExecutorService exec = Executors.newFixedThreadPool(3);

List results = exec.invokeAll(tasks);

// wait for tasks to finish and collect results

int counter = 0;

for(Future f: results) {

counter += f.get();

}