W3AF进程调度相关总结

标签（空格分隔）： web扫描器

---

w3af不同插件间调用采用进程池调度，流程如下：

• 进程池建立
  通过plugins.py的plugin_inst.set_worker_pool(self._w3af_core.worker_pool)创建进程池，其中self._w3af_core.worker_pool调用w3afcore.py的worker_pool()方法，set_worker_pool()为plugin_inst基类plugin.py方法。

• 进程池调用
  通过各个插件的self._send_mutants_in_threads()调用基类plugins.py的 imap_unordered = self.worker_pool.imap_unordered实现进程调度。

• 进程池销毁
  通过Strategy.py的self._w3af_core.worker_pool.finish()销毁进程池。

整体流程图

进程池建立

该模块包括：

• plugins.py的set_worker_pool
• w3afcore.py的worker_pool()
• plugin.py的set_worker_pool()

plugins.py的set_worker_pool

该模块负责框架所有插件初始化、调度等相关工作，进程调度涉及相关代码如下：

    def get_plugin_inst(self, plugin_type, plugin_name):
        """
        :return: An instance of a plugin.
        """
        plugin_inst = factory('w3af.plugins.%s.%s' % (plugin_type, plugin_name))
        plugin_inst.set_url_opener(self._w3af_core.uri_opener)
        plugin_inst.set_worker_pool(self._w3af_core.worker_pool)

        if plugin_name in self._plugins_options[plugin_type].keys():
            custom_options = self._plugins_options[plugin_type][plugin_name]
            plugin_inst.set_options(custom_options)

        # This will init some plugins like mangle and output
        if plugin_type == 'attack' and not self.initialized:
            self.init_plugins()

        return plugin_inst

上述代码中plugin_inst.set_worker_pool(self._w3af_core.worker_pool)进行进程池设置。

w3afcore.py的worker_pool()

该模块是整个框架的核心框架，负责各个插件异常协调调度、线程管理等功能，该模块与进程相关调用的代码如下：

    def worker_pool(self):
        """
        ：构造进程池
        """
        if not hasattr(self, '_worker_pool'):
            # Should get here only on the first call to "worker_pool".
            self._worker_pool = Pool(self.WORKER_THREADS,
                                     worker_names='WorkerThread')

        if not self._worker_pool.is_running():
            self._worker_pool = Pool(self.WORKER_THREADS,
                                     worker_names='WorkerThread')

        return self._worker_pool

worker_pool的作用是构造进程池，供plugins.py模块使用。

plugin.py的set_worker_pool()

该模块为插件基础类，所有插件都继承它。

    def set_worker_pool(self, worker_pool):
        """
        Sets the worker pool (at the moment of writing this is a thread pool)
        that will be used by the plugin to send requests using different
        threads.
        """
        self.worker_pool = worker_pool

进程池调用

这里以audit的os_commanding插件为例进行介绍，相关代码如下：

def _with_echo(self, freq, orig_response):
        # Prepare the strings to create the mutants
        command_list = self._get_echo_commands()
        only_command_strings = [v.get_command() for v in command_list]
        mutants = create_mutants(freq, only_command_strings,
                                 orig_resp=orig_response)

        self._send_mutants_in_threads(self._uri_opener.send_mutant,
                                      mutants,
                                      self._analyze_echo)

上述代码self._send_mutants_in_threads()通过调用其基类plugin.py的_send_mutants_in_threads()实现多进程。

进程池调用销毁

该模块是整个框架的核心策略模块，负责扫描策略的制定，该模块进程线程先关代码如下：
start()方法为strategy.py的入口方法，具体如下：

            ......
            #目标环境检测方法
            self.verify_target_server()
            self._setup_grep()
            self._setup_auth()
            self._setup_crawl_infrastructure()
            #audit插件进程管理方法
            self._setup_audit()
            self._setup_bruteforce()
            self._setup_404_detection()
            self._seed_discovery()
            self._fuzzable_request_router()
            #等待所有的进程结束
            self.join_all_consumers()
            #结束掉w3af_core模块中创建的进程池
            self._w3af_core.worker_pool.finish()

上述代码中_setup_XXX()方法为对应插件管理方法，这里以self._setup_audit()为例介绍，该代码定义如下：

def _setup_audit(self):
        """
        开启audit插件
        """
        om.out.debug('Called _setup_audit()')

        audit_plugins = self._w3af_core.plugins.plugins['audit']

        if audit_plugins:
            self._audit_consumer = audit(audit_plugins, self._w3af_core)
            self._audit_consumer.start()