Python线程池(threading)

ThreadPool.py：

#encoding:utf-8
'''
Created on Mar 28, 2015

@author: root
'''
import threading
import Queue

_pool_size = 15

class AbstractTask:
    '''
    基础任务接口
    '''
    def __init__(self, *args, **kwargs):
        self.args = args
        self.kwargs = kwargs
    
    def execute(self):
        '''
        任务执行函数，必须继承实现
        '''
        raise NotImplementedError

'''
任务队列
'''
tasklist = Queue.Queue(maxsize=_pool_size)
    
class WorkThread(threading.Thread):
    '''
    工作线程
    '''
    def __init__(self, tasklist):
        threading.Thread.__init__(self)
        self.tasklist = tasklist
        self.over = threading.Event()
    
    def run(self):
        while not self.over.is_set():
            try:
                task = self.tasklist.get(True, 3)
            except Queue.Empty:
                continue
                        
            task.execute()
            self.tasklist.task_done()
            
    def dismiss(self):
        self.over.set()

class ThreadPool:
    '''
    线程池
    '''
    def __init__(self, maxsize, tasklist):
        if maxsize < 0:
            raise ValueError("线程的数量必须为大于0")
        
        self.tasklist = tasklist        
        self.workthreads = []
        
        for i in range(maxsize):
            workthread = WorkThread(self.tasklist)
            self.workthreads.append(workthread)
            workthread.start()
        
    def join(self):
        '''
        等待所有线程结束并退出
        '''
        self.tasklist.join()
        for workthread in self.workthreads:
            workthread.dismiss()
        
    def isOver(self):
        '''
        当前是否还有任务在等待执行
        '''
        return self.tasklist.empty()

Demo.py：

#encoding:utf-8
'''
Created on Mar 28, 2015

@author: root
'''
from ThreadPool import AbstractTask, ThreadPool, tasklist
import time, uuid

class MyTask(AbstractTask):
    
    def __init__(self, *args, **kwargs):
        AbstractTask(*args, **kwargs)
        try:
            self.taskname = kwargs['taskname']
        except KeyError:
            self.taskname = str(uuid.uuid4())
            
    def execute(self):
        print self.taskname+" 开始执行"
        time.sleep(3)
        print self.taskname+" 执行结束"

if __name__ == '__main__':
    
    threadpool = ThreadPool(5, tasklist)
    
    for i in range(30):
        task = MyTask()
        tasklist.put(task)
    
    threadpool.join()

在这个例子中用了Queue来做任务队列。Queue中存储的是函数，它只所可以调用join来等待所有线程结束，是因为在每个线程在从任务队列里get一个任务并执行完之后，要调用一下Queue的task_done, 以表明之前get到的任务已经结束了，Queue在调用join的时候就检查一下task_done调用的次数是否等于当前队列中所有任务的数量，如果相等，就说明所有任务都执行完了。

这是Queue的帮助文档：

Help on class Queue in module Queue:

class Queue
 |  Create a queue object with a given maximum size.
 |  
 |  If maxsize is <= 0, the queue size is infinite.
 |  
 |  Methods defined here:
 |  
 |  __init__(self, maxsize=0)
 |  
 |  empty(self)
 |      Return True if the queue is empty, False otherwise (not reliable!).
 |  
 |  full(self)
 |      Return True if the queue is full, False otherwise (not reliable!).
 |  
 |  get(self, block=True, timeout=None)
 |      Remove and return an item from the queue.
 |      
 |      If optional args 'block' is true and 'timeout' is None (the default),
 |      block if necessary until an item is available. If 'timeout' is
 |      a positive number, it blocks at most 'timeout' seconds and raises
 |      the Empty exception if no item was available within that time.
 |      Otherwise ('block' is false), return an item if one is immediately
 |      available, else raise the Empty exception ('timeout' is ignored
 |      in that case).
 |  
 |  get_nowait(self)
 |      Remove and return an item from the queue without blocking.
 |      
 |      Only get an item if one is immediately available. Otherwise
 |      raise the Empty exception.
 |  
 |  join(self)
 |      Blocks until all items in the Queue have been gotten and processed.
 |      
 |      The count of unfinished tasks goes up whenever an item is added to the
 |      queue. The count goes down whenever a consumer thread calls task_done()
 |      to indicate the item was retrieved and all work on it is complete.
 |      
 |      When the count of unfinished tasks drops to zero, join() unblocks.
 |  
 |  put(self, item, block=True, timeout=None)
 |      Put an item into the queue.
 |      
 |      If optional args 'block' is true and 'timeout' is None (the default),
 |      block if necessary until a free slot is available. If 'timeout' is
 |      a positive number, it blocks at most 'timeout' seconds and raises
 |      the Full exception if no free slot was available within that time.
 |      Otherwise ('block' is false), put an item on the queue if a free slot
 |      is immediately available, else raise the Full exception ('timeout'
 |      is ignored in that case).
 |  
 |  put_nowait(self, item)
 |      Put an item into the queue without blocking.
 |      
 |      Only enqueue the item if a free slot is immediately available.
 |      Otherwise raise the Full exception.
 |  
 |  qsize(self)
 |      Return the approximate size of the queue (not reliable!).
 |  
 |  task_done(self)
 |      Indicate that a formerly enqueued task is complete.
 |      
 |      Used by Queue consumer threads.  For each get() used to fetch a task,
 |      a subsequent call to task_done() tells the queue that the processing
 |      on the task is complete.
 |      
 |      If a join() is currently blocking, it will resume when all items
 |      have been processed (meaning that a task_done() call was received
 |      for every item that had been put() into the queue).
 |      
 |      Raises a ValueError if called more times than there were items
 |      placed in the queue.