Python Gearman 分析(1)

python gearman 分析：

偶然看到这个框架，发现他的用途还是挺广泛的，读了他的源码并尝试对其进行一个分析，加深自己的理解同时方便以后其他方面的应用和见解。以下是对python gearman的流程的一个深入的分析。


Server：
1. Server启动并初始化一个TaskManager，之后通过asyncore.loop监控socket连接。
2. 每个新的连接都用GearmanServerClient包装，并且将每个ServerClient连接加入TaskManager管理。
由于Server,ServerClient都继承自asyncore.dispather，他们也将被纳入loop监控体系中。

class GearmanServer(asyncore.dispatcher):
------------------------------------------------------------------
    def __init__(self, host="127.0.0.1", port=DEFAULT_PORT):
        self.manager = GearmanTaskManager()
    def handle_accept(self):
        GearmanServerClient(sock, addr, self, self.manager)
    def start(self):
        while self.running:
            asyncore.loop(timeout=1, use_poll=False, count=1)

class GearmanServerClient(asyncore.dispatcher):
    def __init__(self, sock, addr, server, manager):
        # add self to manager
        manager.register_client(self)


Worker:
1. Worker启动并注册工作函数，然后进入工作状态。
2. 进入工作状态需要以下一些步骤：
1) 检查self.working并循环处理连接
2) 查找alive_connections，如果未连接，尝试连接每个connection并发送本地can_do信息，加入alive并返回列表
3) 查询连接是否有job，如果有则进行处理，否则发送pre_sleep消息并将connection状态置为sleep=True，为防止有新job进入，select connection并确认没有新数据接收，否则将其重新置为sleep=Flase
4) 将没有job的连接暂时休眠

class GearmanWorker(GearmanBaseClient):
------------------------------------------------------------------
    def work(self, stop_if=None, hooks=None):
  (1)   while self.working:
            need_sleep = True
            # Try to grab work from all alive connections
  (2)       for conn in self.alive_connections:
  (3)           worked = self._work_connection(conn, hooks)
                if worked:
                    need_sleep = False
            # If no tasks were handled then sleep and wait for the server
            # to wake us with a 'noop'
  (4)       if need_sleep:
                for conn in alive:
                    if not conn.sleeping:
                        conn.send_command("pre_sleep")
                        conn.sleeping = True


client:
1. Client初始化
2. 提交任务：do_task,do_taskset,dispatch_background_task
3. 查询任务状态或者对处理结果

class GearmanClient(GearmanBaseClient):
------------------------------------------------------------------
    # all three method will use do_taskset to invoke task
    def do_task(self, task):
    def dispatch_background_task(self, func, arg, uniq=None, high_priority=False):
    def do_taskset(self, taskset, timeout=None):
        # set of connections to which jobs were submitted
        taskset.connections = set(self._submit_task(task) for taskset ...)
        while not taskset.cancelled and not all(task.is_finished for ...)
            # select and deal with read write handle
            for conn in rd_list:
                for cmd in conn.recv():
                    self._command_handler(taskset, conn, *cmd)
            for conn in wr_list:
                conn.send()

下图是Gearman各个组件的一个全貌，它们之间的交互都是通过消息传递来实现。