ThreadPool.py:
#encoding:utf-8
'''
Created on Mar 28, 2015
@author: root
'''
import threading
import Queue
_pool_size = 15
class AbstractTask:
'''
基础任务接口
'''
def __init__(self, *args, **kwargs):
self.args = args
self.kwargs = kwargs
def execute(self):
'''
任务执行函数,必须继承实现
'''
raise NotImplementedError
'''
任务队列
'''
tasklist = Queue.Queue(maxsize=_pool_size)
class WorkThread(threading.Thread):
'''
工作线程
'''
def __init__(self, tasklist):
threading.Thread.__init__(self)
self.tasklist = tasklist
self.over = threading.Event()
def run(self):
while not self.over.is_set():
try:
task = self.tasklist.get(True, 3)
except Queue.Empty:
continue
task.execute()
self.tasklist.task_done()
def dismiss(self):
self.over.set()
class ThreadPool:
'''
线程池
'''
def __init__(self, maxsize, tasklist):
if maxsize < 0:
raise ValueError("线程的数量必须为大于0")
self.tasklist = tasklist
self.workthreads = []
for i in range(maxsize):
workthread = WorkThread(self.tasklist)
self.workthreads.append(workthread)
workthread.start()
def join(self):
'''
等待所有线程结束并退出
'''
self.tasklist.join()
for workthread in self.workthreads:
workthread.dismiss()
def isOver(self):
'''
当前是否还有任务在等待执行
'''
return self.tasklist.empty()
Demo.py:
#encoding:utf-8
'''
Created on Mar 28, 2015
@author: root
'''
from ThreadPool import AbstractTask, ThreadPool, tasklist
import time, uuid
class MyTask(AbstractTask):
def __init__(self, *args, **kwargs):
AbstractTask(*args, **kwargs)
try:
self.taskname = kwargs['taskname']
except KeyError:
self.taskname = str(uuid.uuid4())
def execute(self):
print self.taskname+" 开始执行"
time.sleep(3)
print self.taskname+" 执行结束"
if __name__ == '__main__':
threadpool = ThreadPool(5, tasklist)
for i in range(30):
task = MyTask()
tasklist.put(task)
threadpool.join()
在这个例子中用了Queue来做任务队列。Queue中存储的是函数,它只所可以调用join来等待所有线程结束,是因为在每个线程在从任务队列里get一个任务并执行完之后,要调用一下Queue的task_done, 以表明之前get到的任务已经结束了,Queue在调用join的时候就检查一下task_done调用的次数是否等于当前队列中所有任务的数量,如果相等,就说明所有任务都执行完了。
这是Queue的帮助文档:
Help on class Queue in module Queue:
class Queue
| Create a queue object with a given maximum size.
|
| If maxsize is <= 0, the queue size is infinite.
|
| Methods defined here:
|
| __init__(self, maxsize=0)
|
| empty(self)
| Return True if the queue is empty, False otherwise (not reliable!).
|
| full(self)
| Return True if the queue is full, False otherwise (not reliable!).
|
| get(self, block=True, timeout=None)
| Remove and return an item from the queue.
|
| If optional args 'block' is true and 'timeout' is None (the default),
| block if necessary until an item is available. If 'timeout' is
| a positive number, it blocks at most 'timeout' seconds and raises
| the Empty exception if no item was available within that time.
| Otherwise ('block' is false), return an item if one is immediately
| available, else raise the Empty exception ('timeout' is ignored
| in that case).
|
| get_nowait(self)
| Remove and return an item from the queue without blocking.
|
| Only get an item if one is immediately available. Otherwise
| raise the Empty exception.
|
| join(self)
| Blocks until all items in the Queue have been gotten and processed.
|
| The count of unfinished tasks goes up whenever an item is added to the
| queue. The count goes down whenever a consumer thread calls task_done()
| to indicate the item was retrieved and all work on it is complete.
|
| When the count of unfinished tasks drops to zero, join() unblocks.
|
| put(self, item, block=True, timeout=None)
| Put an item into the queue.
|
| If optional args 'block' is true and 'timeout' is None (the default),
| block if necessary until a free slot is available. If 'timeout' is
| a positive number, it blocks at most 'timeout' seconds and raises
| the Full exception if no free slot was available within that time.
| Otherwise ('block' is false), put an item on the queue if a free slot
| is immediately available, else raise the Full exception ('timeout'
| is ignored in that case).
|
| put_nowait(self, item)
| Put an item into the queue without blocking.
|
| Only enqueue the item if a free slot is immediately available.
| Otherwise raise the Full exception.
|
| qsize(self)
| Return the approximate size of the queue (not reliable!).
|
| task_done(self)
| Indicate that a formerly enqueued task is complete.
|
| Used by Queue consumer threads. For each get() used to fetch a task,
| a subsequent call to task_done() tells the queue that the processing
| on the task is complete.
|
| If a join() is currently blocking, it will resume when all items
| have been processed (meaning that a task_done() call was received
| for every item that had been put() into the queue).
|
| Raises a ValueError if called more times than there were items
| placed in the queue.
class Queue
| Create a queue object with a given maximum size.
|
| If maxsize is <= 0, the queue size is infinite.
|
| Methods defined here:
|
| __init__(self, maxsize=0)
|
| empty(self)
| Return True if the queue is empty, False otherwise (not reliable!).
|
| full(self)
| Return True if the queue is full, False otherwise (not reliable!).
|
| get(self, block=True, timeout=None)
| Remove and return an item from the queue.
|
| If optional args 'block' is true and 'timeout' is None (the default),
| block if necessary until an item is available. If 'timeout' is
| a positive number, it blocks at most 'timeout' seconds and raises
| the Empty exception if no item was available within that time.
| Otherwise ('block' is false), return an item if one is immediately
| available, else raise the Empty exception ('timeout' is ignored
| in that case).
|
| get_nowait(self)
| Remove and return an item from the queue without blocking.
|
| Only get an item if one is immediately available. Otherwise
| raise the Empty exception.
|
| join(self)
| Blocks until all items in the Queue have been gotten and processed.
|
| The count of unfinished tasks goes up whenever an item is added to the
| queue. The count goes down whenever a consumer thread calls task_done()
| to indicate the item was retrieved and all work on it is complete.
|
| When the count of unfinished tasks drops to zero, join() unblocks.
|
| put(self, item, block=True, timeout=None)
| Put an item into the queue.
|
| If optional args 'block' is true and 'timeout' is None (the default),
| block if necessary until a free slot is available. If 'timeout' is
| a positive number, it blocks at most 'timeout' seconds and raises
| the Full exception if no free slot was available within that time.
| Otherwise ('block' is false), put an item on the queue if a free slot
| is immediately available, else raise the Full exception ('timeout'
| is ignored in that case).
|
| put_nowait(self, item)
| Put an item into the queue without blocking.
|
| Only enqueue the item if a free slot is immediately available.
| Otherwise raise the Full exception.
|
| qsize(self)
| Return the approximate size of the queue (not reliable!).
|
| task_done(self)
| Indicate that a formerly enqueued task is complete.
|
| Used by Queue consumer threads. For each get() used to fetch a task,
| a subsequent call to task_done() tells the queue that the processing
| on the task is complete.
|
| If a join() is currently blocking, it will resume when all items
| have been processed (meaning that a task_done() call was received
| for every item that had been put() into the queue).
|
| Raises a ValueError if called more times than there were items
| placed in the queue.