插件加载过程以及neutron-server服务端RPC监听建立简析

参考

setuptools：http://yansu.org/2013/06/07/learn-python-setuptools-in-detail.html

stevedore插件加载：http://blog.csdn.net/gqtcgq/article/details/49620279

一、stevedore简要介绍

stevedore是用来实现动态加载代码的开源模块。它是在OpenStack中用来加载插件的公共模块。

stevedore使用setuptools的entry points来定义并加载插件。entry point引用的是定义在模块中的对象，比如类、函数、实例等，只要在import模块时能够被创建的对象都可以。例如neutron-9.0.0-py2.7.egg-info\entry_points.txt（划重点）

在stevedore中，有三种使用插件的方式：Drivers、Hooks、Extensions

二 、Neutron 插件加载

入口neutron/cmd/eventlet/server/__init__.py​的main函数

def main():
    # 启动服务，并调用 _main_neutron_server
    server.boot_server(_main_neutron_server)


def _main_neutron_server():
    # neutron有两种建立api方式，一种是pecan一种是原来的legacy
    if cfg.CONF.web_framework == 'legacy':
        # 这里我们主要讲legacy 下面发析这一步
        wsgi_eventlet.eventlet_wsgi_server()
    else:
        wsgi_pecan.pecan_wsgi_server()


def main_rpc_eventlet(): 

进入函数wsgi_eventlet.eventlet_wsgi_server()：

neutron/server/wsgi_eventlet.py​

def eventlet_wsgi_server():
    #neutron_api是个NeutronApiService对象
    #核心功能一部分是WSGI，另一部分就是rpc部分。这里将Netron提供的API功能封装成了NeutronApiService类
    #----->   会调用service.NeutronApiService的dcreate和start方法
    neutron_api = service.serve_wsgi(service.NeutronApiService)

    # 利用eventlet的线程池启动api和rpc    ------->   启动完wsgi后，启动rpc_workers
    start_api_and_rpc_workers(neutron_api)



#neutron采用的是多进程加GreenPool的并发模型   ---> wsgi app,rpc分别fork不同的子进程来执行，在每个子进程内部通过eventlet提供的GreenPool来提高吞吐量，可以理解为线程池（实际上是GreenThread,协程）
def start_api_and_rpc_workers(neutron_api):
    try:
        #这里是RPC服务相关的东西
        worker_launcher = service.start_all_workers()

        pool = eventlet.GreenPool()

        #从当前的池子中孵化一个可用的greenthread，在这个 greenthread 中执行 function ，参数 *args, **kwargs 为传给 function 的参数。返回一个 GreenThread 对象，这个对象执行着 function ，可以通过该 GreenThread 对象获取 function 的返回值。
        api_thread = pool.spawn(neutron_api.wait)

        # oslo_service\service.py文件中ServiceLauncher              ----->      等待服务退出
        plugin_workers_thread = pool.spawn(worker_launcher.wait)


        # api and other workers should die together. When one dies,
        # kill the other.

        # 主进程中使用GreenPool来运行neutron_api,neutron_rpc的wait方法，并调用waitall方法等待2个GreenThread结束，实际上这意味着主进程只是等待wsgi API,rpc两个子进程结束而已。其中的link方法是确保只要rpc,api有一个服务挂掉就结束另外一个服务。
        api_thread.link(lambda gt: plugin_workers_thread.kill())
        plugin_workers_thread.link(lambda gt: api_thread.kill())

        pool.waitall()
    except NotImplementedError:
        LOG.info(_LI("RPC was already started in parent process by "
                     "plugin."))

        neutron_api.wait()

Rest API我们在《Neutron API建立简析》中有分析过，现在我们这部分是关于RPC API的内容，调用过程为main -->wsgi_eventlet.eventlet_wsgi_server() --> start_api_and_rpc_workers(neutron_api) -->  worker_launcher = service.start_all_workers()

neutron\service.py

def start_all_workers():
    workers = _get_rpc_workers() + _get_plugins_workers()
    return _start_workers(workers)

这是个加载插件的过程，加载过程分别在函数_get_rpc_workers 和 _get_plugins_workers中进行，每个插件被封装为服务，然后在_start_workers中执行。现在我们先看下这个插件加载过程_get_rpc_workers：

neutron\service.py

def _get_rpc_workers():
    #这里获取插件
    #最终会DriverManager
    plugin = manager.NeutronManager.get_plugin()

    service_plugins = (
        manager.NeutronManager.get_service_plugins().values())

    if cfg.CONF.rpc_workers < 1:
        cfg.CONF.set_override('rpc_workers', 1)

    # If 0 < rpc_workers then start_rpc_listeners would be called in a
    # subprocess and we cannot simply catch the NotImplementedError.  It is
    # simpler to check this up front by testing whether the plugin supports
    # multiple RPC workers.

    #这里对插件的实现函数进行测试，如果没有实现则报错
    if not plugin.rpc_workers_supported():
        LOG.debug("Active plugin doesn't implement start_rpc_listeners")
        if 0 < cfg.CONF.rpc_workers:
            LOG.error(_LE("'rpc_workers = %d' ignored because "
                          "start_rpc_listeners is not implemented."),
                      cfg.CONF.rpc_workers)
        raise NotImplementedError()

    # passing service plugins only, because core plugin is among them
    #这里封装成RpcWorker对象
    rpc_workers = [RpcWorker(service_plugins,
                             worker_process_count=cfg.CONF.rpc_workers)]

    if (cfg.CONF.rpc_state_report_workers > 0 and
            plugin.rpc_state_report_workers_supported()):
        rpc_workers.append(
            RpcReportsWorker(
                [plugin],
                worker_process_count=cfg.CONF.rpc_state_report_workers
            )
        )
    return rpc_workers

可以看到第一步加载插件，第二部对插件实现的函数进行测试，如果没有实现的报错处理。第三步，封装成RpcWorker对象。在继续分析之前我们先解释下这个插件是个什么东西。根本上来讲这里的插件是一个抽象类的具体实现，我们摆出抽象类(只贴出部分代码)：

neutron\neutron_plugin_base_v2.py

@six.add_metaclass(abc.ABCMeta)
class NeutronPluginBaseV2(neutron_worker.WorkerSupportServiceMixin):

    @abc.abstractmethod
    def create_subnet(self, context, subnet):
        """Create a subnet.

        Create a subnet, which represents a range of IP addresses
        that can be allocated to devices

        :param context: neutron api request context
        :param subnet: dictionary describing the subnet, with keys
                       as listed in the  :obj:`RESOURCE_ATTRIBUTE_MAP` object
                       in :file:`neutron/api/v2/attributes.py`.  All keys will
                       be populated.
        """
        pass

    .......

    def start_rpc_listeners(self):
        """Start the RPC listeners.

        Most plugins start RPC listeners implicitly on initialization.  In
        order to support multiple process RPC, the plugin needs to expose
        control over when this is started.

        .. note:: this method is optional, as it was not part of the originally
                  defined plugin API.
        """
        raise NotImplementedError()

    def start_rpc_state_reports_listener(self):
        """Start the RPC listeners consuming state reports queue.

        This optional method creates rpc consumer for REPORTS queue only.

        .. note:: this method is optional, as it was not part of the originally
                  defined plugin API.
        """
        raise NotImplementedError()

    def rpc_workers_supported(self):
        """Return whether the plugin supports multiple RPC workers.

        A plugin that supports multiple RPC workers should override the
        start_rpc_listeners method to ensure that this method returns True and
        that start_rpc_listeners is called at the appropriate time.
        Alternately, a plugin can override this method to customize detection
        of support for multiple rpc workers

        .. note:: this method is optional, as it was not part of the originally
                  defined plugin API.
        """
        return (self.__class__.start_rpc_listeners !=
                NeutronPluginBaseV2.start_rpc_listeners)

    def rpc_state_report_workers_supported(self):
        """Return whether the plugin supports state report RPC workers.

        .. note:: this method is optional, as it was not part of the originally
                  defined plugin API.
        """
        return (self.__class__.start_rpc_state_reports_listener !=
                NeutronPluginBaseV2.start_rpc_state_reports_listener)

可以看到就有我们前面看到的rpc_workers_supported和rpc_state_report_workers_supported检测函数，现在回去分析插件加载过程,进入函数NeutronManager.get_plugin()：
neutron\manager.py

   @classmethod
    def get_plugin(cls):
        # Return a weakref to minimize gc-preventing references.
        # Return a weakref to minimize gc-preventing references.
        # weakref不多说，说另外两步，
        # 第一步获取manager实例cls.get_instance()
        # 第二步返回当前的self.plugin

        #对一个对象的弱引用。相对于通常的引用来说，如果一个对象有一个常规的引用，它是不会被垃圾收集器销毁的，但是如果一个对象只剩下一个弱引用，那么它可能被垃圾收集器收回。
        return weakref.proxy(cls.get_instance().plugin)

可以看到直接简单地返回NeutronManager实例成员plugin的弱引用，我们接着看下plugin具体是什么：
neutron\manager.py

@six.add_metaclass(profiler.TracedMeta)
class NeutronManager(object):
    """Neutron's Manager class.

    Neutron's Manager class is responsible for parsing a config file and
    instantiating the correct plugin that concretely implements
    neutron_plugin_base class.
    The caller should make sure that NeutronManager is a singleton.
    """
    _instance = None
    __trace_args__ = {"name": "rpc"}

    def __init__(self, options=None, config_file=None):
        # If no options have been provided, create an empty dict
        if not options:
            options = {}

        msg = validate_pre_plugin_load()
        if msg:
            LOG.critical(msg)
            raise Exception(msg)

        # NOTE(jkoelker) Testing for the subclass with the __subclasshook__
        #                breaks tach monitoring. It has been removed
        #                intentionally to allow v2 plugins to be monitored
        #                for performance metrics.

        plugin_provider = cfg.CONF.core_plugin
        LOG.info(_LI("Loading core plugin: %s"), plugin_provider)

        #这里最终会使用DriverManager加载插件
        self.plugin = self._get_plugin_instance(CORE_PLUGINS_NAMESPACE,
                                                plugin_provider)
        msg = validate_post_plugin_load()
        if msg:
            LOG.critical(msg)
            raise Exception(msg)

        # core plugin as a part of plugin collection simplifies
        # checking extensions
        # TODO(enikanorov): make core plugin the same as
        # the rest of service plugins
        self.service_plugins = {constants.CORE: self.plugin}
        self._load_service_plugins()
        # Used by pecan WSGI
        self.resource_plugin_mappings = {}
        self.resource_controller_mappings = {}
        self.path_prefix_resource_mappings = defaultdict(list)

在NeutronManager的构造函数中的32行，可以看到self.plugin的初始化过程，字面理解是获得插件实例，我们继续分析_get_plugin_instance：

    def _get_plugin_instance(self, namespace, plugin_provider):

        plugin_class = self.load_class_for_provider(namespace, plugin_provider)
        return plugin_class()

接着load_class_for_provider：

   @staticmethod
    def load_class_for_provider(namespace, plugin_provider):
        """Loads plugin using alias or class name
        :param namespace: namespace where alias is defined
        :param plugin_provider: plugin alias or class name
        :returns plugin that is loaded
        :raises ImportError if fails to load plugin
        """

        try:
            return utils.load_class_by_alias_or_classname(namespace,
                    plugin_provider)
        except ImportError:
            raise ImportError(_("Plugin '%s' not found.") % plugin_provider)

接着load_class_by_alias_or_classname：

def load_class_by_alias_or_classname(namespace, name):
    """Load class using stevedore alias or the class name
    :param namespace: namespace where the alias is defined
    :param name: alias or class name of the class to be loaded
    :returns class if calls can be loaded
    :raises ImportError if class cannot be loaded
    """

    if not name:
        LOG.error(_LE("Alias or class name is not set"))
        raise ImportError(_("Class not found."))
    try:
        # Try to resolve class by alias
        #终于到重点了，mgr = driver.DriverManager(namespace, name)这个就是使用stevedore的实现
        mgr = driver.DriverManager(namespace, name)
        class_to_load = mgr.driver
    except RuntimeError:
        e1_info = sys.exc_info()
        # Fallback to class name
        try:
            class_to_load = importutils.import_class(name)
        except (ImportError, ValueError):
            LOG.error(_LE("Error loading class by alias"),
                      exc_info=e1_info)
            LOG.error(_LE("Error loading class by class name"),
                      exc_info=True)
            raise ImportError(_("Class not found."))
    return class_to_load

终于看到真实面目，就是依靠stevedore的driver.DriverManager(namespace, name)，那namespace和name具体是啥？

在文件neutron\manager.py

CORE_PLUGINS_NAMESPACE = 'neutron.core_plugins'

通过参考stevedore和配置文件/etc/neutron/neutron.conf，可以知道加载的插件名为ml2

结合neutron-9.0.0-py2.7.egg-info\entry_points.txt，可知要加载的插件在neutron.plugins.ml2.plugin:Ml2Plugin

[neutron.core_plugins]
ml2 = neutron.plugins.ml2.plugin:Ml2Plugin

总结一下，我们最终得到的是个什么东西。加载插件，最终得到的是一个插件实例，这里的实例就是ml2，也就是neutron.plugins.ml2.plugin:Ml2Plugin实例，然后封装成RpcReportsWorker​对象（函数_get_rpc_workers的返回对象）。

三、运行服务

分析完插件加载，我们分析包含插件的服务的启动过程，回到

def start_all_workers():
    #得到的是插件实例，_get_rpc_workers得到neutron.plugins.ml2.plugin:Ml2Plugin实例
    
    workers = _get_rpc_workers() + _get_plugins_workers()
    #这里会执行插件的RPC监听函数，如ML2的neutron\plugins\ml2\plugin.py
    return _start_workers(workers)

我们进入_start_workers，这里的workers就是前面得到的RpcReportsWorker（划重点）：

def _start_workers(workers):
    process_workers = [
        plugin_worker for plugin_worker in workers
        if plugin_worker.worker_process_count > 0
    ]

    try:
        if process_workers:
            worker_launcher = common_service.ProcessLauncher(
                cfg.CONF, wait_interval=1.0
            )

            # add extra process worker and spawn there all workers with
            # worker_process_count == 0
            #封装成服务
            thread_workers = [
                plugin_worker for plugin_worker in workers
                if plugin_worker.worker_process_count < 1
            ]
             #封装成服务
            if thread_workers:
                process_workers.append(
                    AllServicesNeutronWorker(thread_workers)
                )

            # dispose the whole pool before os.fork, otherwise there will
            # be shared DB connections in child processes which may cause
            # DB errors.
            session.context_manager.dispose_pool()

            #ProcessLauncher的主要作用是根据workers数量来fork不同个数个子进程,再在每个子进程中启动WorkerService。
            for worker in process_workers:
                worker_launcher.launch_service(worker,
                                               worker.worker_process_count)
        else:
            worker_launcher = common_service.ServiceLauncher(cfg.CONF)
            for worker in workers:
                worker_launcher.launch_service(worker)
        return worker_launcher
    except Exception:
        with excutils.save_and_reraise_exception():
            LOG.exception(_LE('Unrecoverable error: please check log for '
                              'details.'))

将插件实例封装成服务实例，生成服务加载器worker_launcher，接着载入服务worker_launcher.launch_service(worker,worker.worker_process_count)：

    #ProcessLauncher的主要作用是根据workers数量来fork不同个数个子进程,再在每个子进程中启动WorkerService。
    def launch_service(self, service, workers=1):
        """Launch a service with a given number of workers.

       :param service: a service to launch, must be an instance of
              :class:`oslo_service.service.ServiceBase`
       :param workers: a number of processes in which a service
              will be running
        """
        _check_service_base(service)
        wrap = ServiceWrapper(service, workers)

        LOG.info(_LI('Starting %d workers'), wrap.workers)

        #通过fork子进程
        #建立一个线程执行self.run_service，函数self.run_service中会执行service.start()
        while self.running and len(wrap.children) < wrap.workers:
            self._start_child(wrap)

接着self._start_child(wrap)

oslo_service\service.py

    def _start_child(self, wrap):
        if len(wrap.forktimes) > wrap.workers:
            # Limit ourselves to one process a second (over the period of
            # number of workers * 1 second). This will allow workers to
            # start up quickly but ensure we don't fork off children that
            # die instantly too quickly.
            if time.time() - wrap.forktimes[0] < wrap.workers:
                LOG.info(_LI('Forking too fast, sleeping'))
                time.sleep(1)

            wrap.forktimes.pop(0)

        wrap.forktimes.append(time.time())
        #子进程
        pid = os.fork()
        if pid == 0:
            #建立一个线程执行self.run_service，函数self.run_service中会执行service.start()
            self.launcher = self._child_process(wrap.service)
            while True: 
                self._child_process_handle_signal()
                status, signo = self._child_wait_for_exit_or_signal(
                    self.launcher)
                if not _is_sighup_and_daemon(signo):
                    self.launcher.wait()
                    break
                self.launcher.restart()

            os._exit(status)

        LOG.debug('Started child %d', pid)

        wrap.children.add(pid)
        self.children[pid] = wrap

        return pid

可以看到会新建子进程，然后在子进程中执行self.launcher = self._child_process(wrap.service)，这实际上是执行service.start()：

oslo_service\service.py

   def _child_process(self, service):
        self._child_process_handle_signal()

        # Reopen the eventlet hub to make sure we don't share an epoll
        # fd with parent and/or siblings, which would be bad
        eventlet.hubs.use_hub()

        # Close write to ensure only parent has it open
        os.close(self.writepipe)
        # Create greenthread to watch for parent to close pipe
        eventlet.spawn_n(self._pipe_watcher)

        # Reseed random number generator
        random.seed()
        #建立一个线程执行self.run_service，函数self.run_service中会执行service.start()
        launcher = Launcher(self.conf, restart_method=self.restart_method)
        launcher.launch_service(service)
        return launcher

继续：

oslo_service\service.py

    #启动服务
    def launch_service(self, service, workers=1):
        """Load and start the given service.

        :param service: The service you would like to start, must be an
                        instance of :class:`oslo_service.service.ServiceBase`
        :param workers: This param makes this method compatible with
                        ProcessLauncher.launch_service. It must be None, 1 or
                        omitted.
        :returns: None

        """
        if workers is not None and workers != 1:
            raise ValueError(_("Launcher asked to start multiple workers"))
        _check_service_base(service)
        service.backdoor_port = self.backdoor_port
        #建立一个线程执行self.run_service，函数self.run_service中会执行service.start()
        self.services.add(service)

继续：

oslo_service\service.py

    def add(self, service):
        """Add a service to a list and create a thread to run it.

        :param service: service to run
        """
        self.services.append(service)

        #建立一个线程执行self.run_service，函数self.run_service中会执行service.start()
        self.tg.add_thread(self.run_service, service, self.done)

继续：

oslo_service\service.py

    @staticmethod
    def run_service(service, done):
        """Service start wrapper.

        :param service: service to run
        :param done: event to wait on until a shutdown is triggered
        :returns: None

        """
        try:
            service.start()
        except Exception:
            LOG.exception(_LE('Error starting thread.'))
            raise SystemExit(1)
        else:
            done.wait()

从前面知道，这里的service就是前面的RpcReportsWorker对象，我们进入RpcReportsWorker：

neutron\service.py

#首先，会根据配置文件core_plugin的配置加载plugin，然后创建RpcWorker，开始监听rpc；通过调用_plugin.start_rpc_listeners进行监听。
class RpcWorker(neutron_worker.NeutronWorker):
    """Wraps a worker to be handled by ProcessLauncher"""
    start_listeners_method = 'start_rpc_listeners'

    def __init__(self, plugins, worker_process_count=1):
        super(RpcWorker, self).__init__(
            worker_process_count=worker_process_count
        )

        self._plugins = plugins
        self._servers = []

    def start(self):
        super(RpcWorker, self).start()
        for plugin in self._plugins:
            if hasattr(plugin, self.start_listeners_method):
                try:
                    #开始RPC监听
                    servers = getattr(plugin, self.start_listeners_method)()
                except NotImplementedError:
                    continue
                self._servers.extend(servers)

    def wait(self):
        try:
            self._wait()
        except Exception:
            LOG.exception(_LE('done with wait'))
            raise

    def _wait(self):
        LOG.debug('calling RpcWorker wait()')
        for server in self._servers:
            if isinstance(server, rpc_server.MessageHandlingServer):
                LOG.debug('calling wait on %s', server)
                server.wait()
            else:
                LOG.debug('NOT calling wait on %s', server)
        LOG.debug('returning from RpcWorker wait()')

    def stop(self):
        LOG.debug('calling RpcWorker stop()')
        for server in self._servers:
            if isinstance(server, rpc_server.MessageHandlingServer):
                LOG.debug('calling stop on %s', server)
                server.stop()

    @staticmethod
    def reset():
        config.reset_service()

可以看到start函数中，先得到插件neutron.plugins.ml2.plugin:Ml2Plugin的成员start_rpc_listeners函数，然后执行该函数：

neutron\plugins\ml2\plugin.py

  @log_helpers.log_method_call
    def start_rpc_listeners(self):
        """Start the RPC loop to let the plugin communicate with agents."""
        self._setup_rpc()
        self.topic = topics.PLUGIN
        self.conn = n_rpc.create_connection()
        self.conn.create_consumer(self.topic, self.endpoints, fanout=False)
        self.conn.create_consumer(
            topics.SERVER_RESOURCE_VERSIONS,
            [resources_rpc.ResourcesPushToServerRpcCallback()],
            fanout=True)
        # process state reports despite dedicated rpc workers
        self.conn.create_consumer(topics.REPORTS,
                                  [agents_db.AgentExtRpcCallback()],
                                  fanout=False)
        return self.conn.consume_in_threads()

可以看到跟RPC建立相关的东西，我们这里不展开。