q.task_done(),每次从queue中get一个数据之后,当处理好相关问题,最后调用该方法,以提示q.join()是否停止阻塞,让线程向前执行或者退出;
如果把最后join部分改变顺序,即改为
out_queue.join()
queue.join()
则程序的运行必然会出现意外,即main()还未执行完,就开始调用
print "Elapsed Time: %s" % (time.time() - start)
比如某次运行结果:
>>> ================================ RESTART ================================
>>>
[<title>Apple</title>]
[<title>IBM - United States</title>]
[<title>Google</title>]
[<title>Amazon.com: Online Shopping for Electronics, Apparel, Computers, Books, DVDs & more</title>]
Elapsed Time: 28.2170000076
>>> [<title>Yahoo!</title>]
...且此处线程未正常退出
q.join(),阻塞,直到queue中的数据均被删除或者处理。为队列中的每一项都调用一次。
对于生产者-消费者模型,这样做还是有问题的,因为如果queue初始为空,q.join()会直接停止阻塞,继而执行后续语句;
如果有多个消费者,没有生产者,且queue始初化为一定的数据量,则可以正常执行。
更进一步,有如下代码:引自 http://www.ibm.com/developerworks/cn/aix/library/au-threadingpython/
#!/usr/bin/env python
import Queue
import threading
import urllib2
import time
from BeautifulSoup import BeautifulSoup
hosts = ["http://yahoo.com", "http://google.com", "http://amazon.com",
"http://ibm.com", "http://apple.com"]
queue = Queue.Queue()
out_queue = Queue.Queue()
class ThreadUrl(threading.Thread):
"""Threaded Url Grab"""
def __init__(self, queue, out_queue):
threading.Thread.__init__(self)
self.queue = queue
self.out_queue = out_queue
def run(self):
while True:
#grabs host from queue
host = self.queue.get()
#grabs urls of hosts and then grabs chunk of webpage
url = urllib2.urlopen(host)
chunk = url.read()
#place chunk into out queue
self.out_queue.put(chunk)
#signals to queue job is done
self.queue.task_done()
class DatamineThread(threading.Thread):
"""Threaded Url Grab"""
def __init__(self, out_queue):
threading.Thread.__init__(self)
self.out_queue = out_queue
def run(self):
while True:
#grabs host from queue
chunk = self.out_queue.get()
#parse the chunk
soup = BeautifulSoup(chunk)
print soup.findAll(['title'])
#signals to queue job is done
self.out_queue.task_done()
start = time.time()
def main():
#spawn a pool of threads, and pass them queue instance
for i in range(5):
t = ThreadUrl(queue, out_queue)
t.setDaemon(True)
t.start()
#populate queue with data
for host in hosts:
queue.put(host)
for i in range(5):
dt = DatamineThread(out_queue)
dt.setDaemon(True)
dt.start()
#wait on the queue until everything has been processed
queue.join()
out_queue.join()
main()
print "Elapsed Time: %s" % (time.time() - start)
如果把最后join部分改变顺序,即改为
out_queue.join()
queue.join()
则程序的运行必然会出现意外,即main()还未执行完,就开始调用
print "Elapsed Time: %s" % (time.time() - start)
比如某次运行结果:
>>> ================================ RESTART ================================
>>>
[<title>Apple</title>]
[<title>IBM - United States</title>]
[<title>Google</title>]
[<title>Amazon.com: Online Shopping for Electronics, Apparel, Computers, Books, DVDs & more</title>]
Elapsed Time: 28.2170000076
>>> [<title>Yahoo!</title>]
...且此处线程未正常退出