java 迭代 线程,Java多线程和迭代器，应该简单，初学者

First I'd like to say that I'm working my way up from python to more complicated code. I'm now on to Java and I'm extremely new. I understand that Java is really good at multithreading which is good because I'm using it to process terabytes of data.

The data input is simply input into an iterator and I have a class that encapsulates a run function that takes one line from the iterator, does some analysis, and then writes the analysis to a file. The only bit of info the threads have to share with each other is the name of the object they are writing to. Simple right? I just want each thread executing the run function simultaneously so we can iterate through the input data quickly. In python it would b e simple.

from multiprocessing import Pool

f = open('someoutput.csv','w');

def run(x):

f.write(analyze(x))

p = Pool(8);

p.map(run,iterator_of_input_data);

So in Java, I have my 10K lines of analysis code and can very easily iterate through my input passing it my run function which in turn calls on all my analysis code sending it to an output object.

public class cool {

...

public static void run(Input input,output) {

Analysis an = new Analysis(input,output);

}

public static void main(String args[]) throws Exception {

Iterator iterator = new Parser(File(input_file)).iterator();

File output = File(output_object);

while(iterator.hasNext(){

cool.run(iterator.next(),output);

}

}

}

All I want to do is get multiple threads taking the iterator objects and executing the run statement. Everything is independent. I keep looking at java multithreading stuff but its for talking over networks, sharing data etc. Is this is simple as I think it is? If someone can just point me in the right direction I would be happy to do the leg work.

thanks

解决方案

A ExecutorService (ThreadPoolExecutor) would be the Java equivelant.

ExecutorService executorService =

new ThreadPoolExecutor(

maxThreads, // core thread pool size

maxThreads, // maximum thread pool size

1, // time to wait before resizing pool

TimeUnit.MINUTES,

new ArrayBlockingQueue(maxThreads, true),

new ThreadPoolExecutor.CallerRunsPolicy());

ConcurrentLinkedQueue resultQueue;

while (iterator.hasNext()) {

executorService.execute(new MyJob(iterator.next(), resultQueue))

}

Implement your job as a Runnable.

class MyJob implements Runnable {

/* collect useful parameters in the constructor */

public MyJob(...) {

/* omitted */

}

public void run() {

/* job here, submit result to resultQueue */

}

}

The resultQueue is present to collect the result of your jobs.

See the java api documentation for detailed information.