pyspider_analysis_3

pyspider.run.py中click框架分析

• run.py以click为整体框架
  • run.py中将各个command与cli关联

    @click.group(invoke_without_command=True)
    ...
    def cli(ctx, **kwargs):
        ...
    
    @cli.command()  # 关联到cli 
    ...
    def scheduler(ctx, xmlrpc, xmlrpc_host, xmlrpc_port,
        inqueue_limit, delete_time, active_tasks, loop_limit, fail_pause_num,
        scheduler_cls, threads, get_object=False):
        ...
    
        ```
    

• 可参考转载文章click的相关使用
  • click官方文档
  • 注意，若click.option中含有callback参数，则该option对应的value不会传到被装饰函数的kwarg中，直接交由callback对象处理， 需要由callback函数中使用return将结果返回给kwarg
    如

    def test_call(ctx, parm, value):
        print(value)
    
    @click.command()
    @click.option("--test", envvar="Path", help="Test", callback=test_call)
    @click.pass_context
    def test(ctx, **kwargs):
        ctx.obj = [1]
        click.echo(ctx)
        click.echo(kwargs)
    
    if __name__ == "__main__":
        test()
    

    测试

    python click_test.py --test 1
    

    结果为
    

• click.group中invoke_without_command参数的使用

  • invoke_without_command = True，则在命令行启动时可不输入被装饰函数的具体command，直接使用[OPTION] [ARGS]的形式，而即使启动了其他command，被修饰函数所代表的command依旧会自动启动

    
    

    def test_call(ctx, parm, value):
        print(value)
    
    @click.group(invoke_without_command=True)
    @click.option("--cli")
    @click.pass_context
    def cli(ctx, **kwargs):
        print("Need True to invoke {}".format(kwargs))
        click.echo(ctx.invoked_subcommand)
        if ctx.invoked_subcommand == "test":
            print("test starts")
    
    @cli.command()
    @click.option("--test", envvar="Path", help="Test", callback=test_call)
    @click.pass_context
    def test(ctx, **kwargs):
        ctx.obj = []
        click.echo(kwargs)
    
    
    if __name__ == "__main__":
        cli()
    

    测试

    python click_test.py --cli group
    python click_test.py test
    

    结果为
    
    

    
    

  • 反之不再赘述

  
  

• 至此，根据run.py中的代码片段

  @click.group(invoke_without_command=True)
  ...
  def cli(ctx, **kwargs):
      ...
      if ctx.invoked_subcommand is None and not ctx.obj.get('testing_mode'):
          ctx.invoke(all)
      ...
  

  可知，一般情况下在命令行启动pyspider时，输入pyspider和pyspider all的效果是一样的

• run.py中的click.pass_context

  • click.pass_context用于command间的消息传递
    如

    def test_call(ctx, parm, value):
        print("callback:", value)
    
    @click.group(invoke_without_command=True)
    @click.option("--cli")
    @click.pass_context
    def cli(ctx, **kwargs):
        ctx.obj = dict()
        ctx.obj['passer'] = kwargs
    
    @cli.command()
    @click.option("--test", envvar="Path", help="Test", callback=test_call)
    @click.pass_context
    def test(ctx, **kwargs):
        print("obj:", ctx.obj)
    
    if __name__ == "__main__":
        cli()
    

    测试

    python click_test.py --cli pokemon test --test pass
    

    结果
    

  
  

  • ctx的type为click.core.Context object， 使用ctx.__dict__查看其具体内容，内容举例如下

    {   'parent': None, 
        'command': <click.core.Group object at 0x00000295428C3E10>,
        'info_name': 'click_test.py', 
        'params': {'cli': None}, 
        'args': [], 
        'protected_args': [], 
        'obj': {'passer': {'cli': None}}, 
        '_meta': {}, 
        'default_map': {'GF': 'Pokemon'}, 
        'invoked_subcommand': None, 
        'terminal_width': None, 
        'max_content_width': None, 
        'allow_extra_args': True, 
        'allow_interspersed_args': False, 
        'ignore_unknown_options': False, 
        'help_option_names': ['--help'], 
        'token_normalize_func': None, 
        'resilient_parsing': False, 
        'auto_envvar_prefix': None, 
        'color': None, 
        '_close_callbacks': [], 
        '_depth': 2
    }
    

    其中params对应[OPTIONS] [ARGS]
default_map用来代替click.option中default，其官方解释为

    By default, the default value for a parameter is pulled from the default flag that is provided when it’s defined, but that’s not the only place defaults can be loaded from. The other place is the Context.default_map (a dictionary) on the context. This allows defaults to be loaded from a configuration file to override the regular defaults.

    官方例子

    import click
    
    @click.group()
    def cli():
        pass
    
    @cli.command()
    @click.option('--port', default=8000)
    def runserver(port):
        click.echo('Serving on http://127.0.0.1:%d/' % port)
    
    if __name__ == '__main__':
        cli(default_map={
            'runserver': {
                'port': 5000
            }
        })
    

    结果为

    $ cli runserver
    Serving on http://127.0.0.1:5000/
    

    
    

    我的例子

    def test_call(ctx, parm, value):
        return value
    
    @click.group(invoke_without_command=True)
    @click.option("--cli")
    @click.pass_context
    def cli(ctx, **kwargs):
        ctx.obj = dict()
        ctx.obj['passer'] = kwargs
        ctx.default_map = {"test": {"test": 1}}
    
    @cli.command()
    @click.option("-t", "--test", default=3, help="Test", callback=test_call)
    @click.pass_context
    def test(ctx, **kwargs):
        click.echo(kwargs)
    
    
    if __name__ == "__main__":
        cli()
    
    

    cmd中输入

    python click_test.py test
    

    结果为
    

    
    
    

    invoked_subcommand为已经启动的命令的名字
    更多用法可参考官方文档中commands的相关部分

  
  

  • 在pyspider中ctx.obj申明为

    ctx.obj = utils.ObjectDict(ctx.obj or {})
    

    
    

    在pyspider.libs.utils中ObjectDict具体实现为

    class ObjectDict(dict):
    """
    Object like dict, every dict[key] can visite by dict.key
    
    If dict[key] is `Get`, calculate it's value.
    """
    
    def __getattr__(self, name):
        ret = self.__getitem__(name)
        if hasattr(ret, '__get__'):
            return ret.__get__(self, ObjectDict)
        return ret
    

    
    

    即传递的ctx.obj为类字典类型的数据结构，可通过dict.key的方式访问key对应的value

    
    

  • 其中__getattr__的作用是当使用点号获取实例属性时，如果属性不存在就自动调用__getattr__方法，可迭代对象的__getitem__用于查找索引对应的值
    如

    class TestObject:
        def __init__(self):
            self.test = 1
    
        def __getattr__(self, item):
            return item
    
    
    if __name__ == "__main__":
        object_test = TestObject()
        dict_test = {"test": 1}
        li_test = [1, 2, 3]
        str_test = "pokemon"
        print("object: ", object_test.__getattribute__("test"))
        print("object has no such attribute:", object_test.pokemon)
        print("dict: ", dict_test.__getitem__("test"))
        print("list: ", li_test.__getitem__(1))
        print("str: ", str_test.__getitem__(0))
    

    结果为
    

  • 关于__get__的相关解释可参考转载文章python set get 等解释