python concurrent queue,如何在python 3中将队列与将来的并发ThreadPoolExecutor一起使用？...

I am using simple threading modules to do concurrent jobs. Now I would like to take advantages of concurrent futures modules. Can some put me a example of using a queue with concurrent library?

I am getting TypeError: 'Queue' object is not iterable

I dont know how to iterate queues

code snippet:

def run(item):

self.__log.info(str(item))

return True

with concurrent.futures.ThreadPoolExecutor(max_workers = 100) as executor:

furtureIteams = { executor.submit(run, item): item for item in list(queue)}

for future in concurrent.futures.as_completed(furtureIteams):

f = furtureIteams[future]

print(f)

解决方案

I would suggest something like this:

def run(queue):

item = queue.get()

self.__log.info(str(item))

return True

workerThreadsToStart = 10

with concurrent.futures.ThreadPoolExecutor(max_workers = 100) as executor:

furtureIteams = { executor.submit(run, queue): index for intex in range(workerThreadsToStart)}

for future in concurrent.futures.as_completed(furtureIteams):

f = furtureIteams[future]

print(f)

The problem you will run in is that a queue is thought to be endless and as a medium to decouple the threads that put something into the queue and threads that get items out of the queue.

When

you have a finite number of items or

you compute all items at once

and afterwards process them in parallel, a queue makes no sense.

A ThreadPoolExecutor makes a queue obsolete in these cases.

I had a look at the ThreadPoolExecutor source:

def submit(self, fn, *args, **kwargs): # line 94

self._work_queue.put(w) # line 102

A Queue is used inside.