Neutron 软件架构设计与实现

Neutron 的软件架构设计

1. Neutron 作为一个分布式系统，采用了主从分布式架构设计，具有负责在中央控制的 Neutron Server 组件（接收北向 API 请求，控制逻辑，下达任务），也有负责在地方执行的 Agents 组件（执行任务，反馈结果）。Neutron Server 和 Agents 之间采用 MQ/RPC 异步通讯模型。

2. Neutron 作为一个开源的 Network as a Service 项目，首先需要考虑能够兼容多种不同的 Network Providers（底层网络技术），例如：Linux Bridge、Open vSwitch、SDN Controller 等等。所以 Neutron Server 还采用了 Multi-Plugins（多插件类型）架构设计，与 Agents 一一对应。

Neutron API

Neutron API 由 Web Server 和 WSGI Application 两部分组成：

1. Web Server：Neutron Server 首先是一个 Web Server，可以对外提供基于 HTTP 协议的 RESTful API。

2. WSGI Application：WSGI 是 Python Application 和 Web Server 进行集成的标准。

在 Neutron API 层面，考虑到 Services（服务类型）的可扩展性，可以进一步细分为 2 类 API：

• Core API：用于提供标准的网络服务，对应 Core Plugins，包括：社区提供的 ML2 Plugin，或者是 3-party 提供的 Vendor Plugins。向上提供标准的 Network、Subnet、Port API 资源类型。

• Resource and Attribute Extension API：用于提供扩展的网络服务，对应 Service Plugins，包括：社区提供的 Routing、LB、VPN、FW 等 Plugins。向上提供 Router、LBaaS、VPNaaS、FWaaS 等 API 资源类型。

Web Server 和 WSGI Application

Neutron Server 对应的服务进程是 neutron-server.service，包含了 Web Server、WSGI Applicant、Plugins、RPC、DB ORM等功能模块。

neutron-server.service 的程序入口函数：

# /opt/stack/neutron/neutron/cmd/eventlet/server/__init__.py
def main():
    server.boot_server(wsgi_eventlet.eventlet_wsgi_server)

# /opt/stack/neutron/neutron/server/wsgi_eventlet.py
def eventlet_wsgi_server():
    # 获取 Neutron Server API WSGI Application
    neutron_api = service.serve_wsgi(service.NeutronApiService)
    # 启动 API 和 RPC 服务
    start_api_and_rpc_workers(neutron_api)

Web Server 通过加载 WSGI Module 将 Web Server 划分为 WSGI Server、WSGI Middleware 和 WSGI Application。我们主要关注 Neutron Server 的 WSGI Application。关键代码如下：

# /opt/stack/neutron/neutron/common/config.py

def load_paste_app(app_name):
    """Builds and returns a WSGI app from a paste config file.

    :param app_name: Name of the application to load
    """
    # oslo_service 的封装函数，解析 api-paste.ini 文件并加载其定义的 Apps 实例
    loader = wsgi.Loader(cfg.CONF)

    # Log the values of registered opts
    if cfg.CONF.debug:
        cfg.CONF.log_opt_values(LOG, logging.DEBUG)
        
    # 实参 app_name 为 `neutron`
    app = loader.load_app(app_name)
    return app

经过Paste库（一套Python的Web开发工具）的一系列处理后，程序流进入pipline_factory函数。

/opt/stack/neutron/neutron/pecan_wsgi/app.py
def v2_factory(global_config, **local_config):
    # REST API 的根 "/" 控制器是 root.V2Controller
    app = pecan.make_app(root.V2Controller(),
                         debug=False,
                         force_canonical=False,
                         hooks=app_hooks,
                         guess_content_type_from_ext=True)
    # 初始化 Neutron Server
    startup.initialize_all()
    return app

至此就得到了一个完整的 Neutron Server 服务进程，同时我们也找到了 API Request 的 "/" 入口。

Core API 和 Extension API

Neutron Server 启动后我们可以通过 curl 获取到当前支持的 Core API 和 Extension API 清单。

Neutron Multi-Plugins

与 Neutron API 对应的，Neutron Multi-Plugins 也有 Core Plugins 和 Service Plugins 这两大类型。可以通过编辑 neutron.conf 指定你希望启动的 Core 和 Service Plugins 的具体类型。

通常的，Core Plugin 选择使用 ML2，而 Service Plugins 这可以指定多个，包括：L3RouterPlugin、FWaaSPlugin、LBaaSPlugin、VPNaaSPlugin 等等。

[DEFAULT]
...
# The core plugin Neutron will use (string value)
core_plugin = ml2

# The service plugins Neutron will use (list value)
service_plugins = neutron.services.l3_router.l3_router_plugin.L3RouterPlugin

Neutron Server 加载对应的 Plugins 代码的实现如下：

# /opt/stack/neutron/neutron/manager.py


class NeutronManager(object):
    """Neutron's Manager class.

    Neutron's Manager class is responsible for parsing a config file and
    instantiating the correct plugin that concretely implements
    neutron_plugin_base class.
    """
...
    def __init__(self, options=None, config_file=None):
    ...
        # 从配置文件中读取 Core Plugin，e.g. ml2
        plugin_provider = cfg.CONF.core_plugin
        LOG.info("Loading core plugin: %s", plugin_provider)
        # NOTE(armax): keep hold of the actual plugin object

        # 实例化 Core Plugin Class，e.g. ML2Plugin Class
        plugin = self._get_plugin_instance(CORE_PLUGINS_NAMESPACE,
                                           plugin_provider)
        
        # 将 Core Plugin 登记到 Plugins Directory
        directory.add_plugin(lib_const.CORE, plugin)
        ...
        
        # load services from the core plugin first
        # 首先加载 Core Plugin 本身默认支持的 Extension Plugins
        self._load_services_from_core_plugin(plugin)
        
        # 继续加载通过配置文件额外指定的 Extension Plugins
        self._load_service_plugins()
        ...

    def _load_services_from_core_plugin(self, plugin):
        """Puts core plugin in service_plugins for supported services."""
        LOG.debug("Loading services supported by the core plugin")

        # supported service types are derived from supported extensions
        # 来之 Core Plugin 默认支持的 Extension Plugins（e.g. lbaas、fwaas、vpnaas、router、qos）
        for ext_alias in getattr(plugin, "supported_extension_aliases", []):
            if ext_alias in constants.EXT_TO_SERVICE_MAPPING:
                service_type = constants.EXT_TO_SERVICE_MAPPING[ext_alias]

                # 将 Extension Plugins 登记到 Plugins Directory
                directory.add_plugin(service_type, plugin)
                LOG.info("Service %s is supported by the core plugin",
                         service_type)

    def _load_service_plugins(self):
        """Loads service plugins.

        Starts from the core plugin and checks if it supports
        advanced services then loads classes provided in configuration.
        """
        # 从配置文件中读取 Extension Plugins
        plugin_providers = cfg.CONF.service_plugins
        
        # 获取 Core Plugin 使用 Neutron 原生的 Datastore 则会返回一些默认的 Extension Plugins（e.g. tag、timestamp、flavors、revisions）
        plugin_providers.extend(self._get_default_service_plugins())
        
        LOG.debug("Loading service plugins: %s", plugin_providers)
        for provider in plugin_providers:
            if provider == '':
                continue

            LOG.info("Loading Plugin: %s", provider)
            # 实例化 Extension Plugins Class 的对象
            plugin_inst = self._get_plugin_instance('neutron.service_plugins',
                                                    provider)

            # only one implementation of svc_type allowed
            # specifying more than one plugin
            # for the same type is a fatal exception
            # TODO(armax): simplify this by moving the conditional into the
            # directory itself.
            plugin_type = plugin_inst.get_plugin_type()
            if directory.get_plugin(plugin_type):
                raise ValueError(_("Multiple plugins for service "
                                   "%s were configured") % plugin_type)

            # 将 Extension Plugins 登记到 Plugins Directory
            directory.add_plugin(plugin_type, plugin_inst)

            # search for possible agent notifiers declared in service plugin
            # (needed by agent management extension)
            # 获取 Core Plugin 实例对象
            plugin = directory.get_plugin()
            if (hasattr(plugin, 'agent_notifiers') and
                    hasattr(plugin_inst, 'agent_notifiers')):
                # 将 Extension Plugins 对应的 Agent notifiers 更新到 Core Plugin 的 Agent notifiers 字典中
                plugin.agent_notifiers.update(plugin_inst.agent_notifiers)

            # disable incompatible extensions in core plugin if any
            utils.disable_extension_by_service_plugin(plugin, plugin_inst)

至此，Neutron 就将当前选择使用的 Core Plugin 和 Extension Plugin 都被实例化并登记到 Plugins Directory 了。

Plugins 注册完成后，会在启动 neutron-server.service 服务进程的过程中同步启动对应的 Plugin Workers 进程或协程。前面提到 Neutron Plugins 和对应的 Neutron Agents 通过 MQ/RPC 来进行通信，所以 Plugin Workers 具有 2 种类型：

• RpcWorker：核心业务逻辑的 RPC Worker。可以通过 neutron.conf 中的配置项 rpc_workers 来设定 RPC Worker 的数量。

• RpcReportsWorker：仅具有 RPC State 的 Report 功能。可以通过 neutron.conf 中的配置项 rpc_state_report_workers 和 rpc_state_report_workers 来指定是否要开启 RPC State Report 功能，以及 Workers 的数量。

# /opt/stack/neutron/neutron/server/wsgi_eventlet.py

# 在启动 neutron-server.servce 时执行
def start_api_and_rpc_workers(neutron_api):
    try:
        # 获取所有 RPC and Plugins workers 的 Launcher 实例对象
        worker_launcher = service.start_all_workers()

        # 创建协程池
        pool = eventlet.GreenPool()
        # 以协程的方式启动 WSGI Application 
        api_thread = pool.spawn(neutron_api.wait)
        # 以协程的方式启动 RPC and Plugins workers
        plugin_workers_thread = pool.spawn(worker_launcher.wait)

        # api and other workers should die together. When one dies,
        # kill the other.
        api_thread.link(lambda gt: plugin_workers_thread.kill())
        plugin_workers_thread.link(lambda gt: api_thread.kill())

        pool.waitall()
    except NotImplementedError:
        LOG.info("RPC was already started in parent process by "
                 "plugin.")

        neutron_api.wait()


# /opt/stack/neutron/neutron/service.py
def _get_rpc_workers():
    # 从 Plugins Directory 获取 Core plugin 实例对象
    plugin = directory.get_plugin()
    # 从 Plugins Directory 获取 Service plugins（包括 Core Plugin）清单列表
    service_plugins = directory.get_plugins().values()
...
    # passing service plugins only, because core plugin is among them
    # 创建所有 Plugins（core + service）的 RpcWorker 对象实例
    rpc_workers = [RpcWorker(service_plugins,
                             worker_process_count=cfg.CONF.rpc_workers)]

    if (cfg.CONF.rpc_state_report_workers > 0 and
            plugin.rpc_state_report_workers_supported()):
        rpc_workers.append(
            RpcReportsWorker(
                [plugin],
                worker_process_count=cfg.CONF.rpc_state_report_workers
            )
        )
    return rpc_workers


class RpcWorker(neutron_worker.BaseWorker):
    """Wraps a worker to be handled by ProcessLauncher"""
    start_listeners_method = 'start_rpc_listeners'

    def __init__(self, plugins, worker_process_count=1):
        super(RpcWorker, self).__init__(
            worker_process_count=worker_process_count
        )

        self._plugins = plugins
        self._servers = []

    def start(self):
        super(RpcWorker, self).start()
        for plugin in self._plugins:
            if hasattr(plugin, self.start_listeners_method):
                try:
                    # 获取所有 Plugins 的 RPC listeners 方法
                    servers = getattr(plugin, self.start_listeners_method)()
                except NotImplementedError:
                    continue
                self._servers.extend(servers)
    ...


class RpcReportsWorker(RpcWorker):
    start_listeners_method = 'start_rpc_state_reports_listener'

通过API Controllers 来关联 API 和 Plugin

在 Pecan 框架中，Controller 是 API 的具体实现，包含了 WSGI Application 中要求的 URL Routed、View Function、HTTP Method 以及三者间 Mapper 的封装。这里我们主要关注 Controller 是如何实现的。

在 neutronapiapp_v2_0 Factory Function 的代码中，除了生成并返回 WSGI Application 对象之外，还执行了 startup.initialize_all()，它所做的事情就是准备好 neutron-server.service 启动所必须的前提条件。包括：加载 Plugins 代码、实例化 API Controllers、以及处理 API、Controllers、Plugins 这 3 者之间的 Mapping 关系。

下述代码中 initialize_all() 所做的事情就是首先加载所有选择使用的 Plugins 代码，然后将 Core Plugins & Core API、Extension Plugins & Extension API 重新封装为一个 CollectionsController 实例对象，并统一注册到 NeutronManger 实例属性self.resource_plugin_mappings 和 self.resource_controller_mappings 中。

# /opt/stack/neutron/neutron/pecan_wsgi/startup.py

# Core Resources 清单
RESOURCES = {'network': 'networks',
             'subnet': 'subnets',
             'subnetpool': 'subnetpools',
             'port': 'ports'}


def initialize_all():
    # 加载 Plugins，如上文所说
    manager.init()
    # PluginAwareExtensionManager 做的事情就是从 configured extension path 加载 extensions，并且对 Extension Plugins 提供一些常用管理函数，e.g. add_extension、extend_resources、get_resources
    ext_mgr = extensions.PluginAwareExtensionManager.get_instance()
    # 将 Core Resources 加入到了 Extension Resources 清单
    ext_mgr.extend_resources("2.0", attributes.RESOURCES)
    # At this stage we have a fully populated resource attribute map;
    # build Pecan controllers and routes for all core resources
    # 获取 Core Plugin 实例对象
    plugin = directory.get_plugin()

    # 循环处理 Core Resources
    for resource, collection in RESOURCES.items():
        # Keeps track of Neutron resources for which quota limits are enforced.
        resource_registry.register_resource_by_name(resource)
        # 将 Core Resource、Core Plugin 封装到 new_controller 实例对象
        new_controller = res_ctrl.CollectionsController(collection, resource,
                                                        plugin=plugin)
        # 将 new_controller 以 resource_name:new_controller 的方式保存到 NeutronManager 实例属性
        manager.NeutronManager.set_controller_for_resource(
            collection, new_controller)
        # 将 plugin 以 resource_name:plugin 的方式保存到 NeutronManager 实例属性
        manager.NeutronManager.set_plugin_for_resource(collection, plugin)

    pecanized_resources = ext_mgr.get_pecan_resources()
    for pec_res in pecanized_resources:
        manager.NeutronManager.set_controller_for_resource(
            pec_res.collection, pec_res.controller)
        manager.NeutronManager.set_plugin_for_resource(
            pec_res.collection, pec_res.plugin)

    # Now build Pecan Controllers and routes for all extensions
    # 获取 Extension Resources 对应的 ResourceExtension 
    resources = ext_mgr.get_resources()
    # Extensions controller is already defined, we don't need it.
    resources.pop(0)
    # 循环处理 ResourceExtension objects
    for ext_res in resources:
        path_prefix = ext_res.path_prefix.strip('/')
        collection = ext_res.collection
        # Retrieving the parent resource.  It is expected the format of
        # the parent resource to be:
        # {'collection_name': 'name-of-collection',
        #  'member_name': 'name-of-resource'}
        # collection_name does not appear to be used in the legacy code
        # inside the controller logic, so we can assume we do not need it.
        parent = ext_res.parent or {}
        parent_resource = parent.get('member_name')
        collection_key = collection
        if parent_resource:
            collection_key = '/'.join([parent_resource, collection])
        collection_actions = ext_res.collection_actions
        member_actions = ext_res.member_actions
        if manager.NeutronManager.get_controller_for_resource(collection_key):
            # This is a collection that already has a pecan controller, we
            # do not need to do anything else
            continue
        legacy_controller = getattr(ext_res.controller, 'controller',
                                    ext_res.controller)
        new_controller = None
        if isinstance(legacy_controller, base.Controller):
            resource = legacy_controller.resource
            plugin = legacy_controller.plugin
            attr_info = legacy_controller.attr_info
            member_actions = legacy_controller.member_actions
            pagination = legacy_controller.allow_pagination
            sorting = legacy_controller.allow_sorting
            # NOTE(blogan): legacy_controller and ext_res both can both have
            # member_actions.  the member_actions for ext_res are strictly for
            # routing, while member_actions for legacy_controller are used for
            # handling the request once the routing has found the controller.
            # They're always the same so we will just use the ext_res
            # member_action.
            # 将 Extension Resource、Extension Plugin、原 Extension Controller 的部分属性重新封装到 new_controller 实例对象
            new_controller = res_ctrl.CollectionsController(
                collection, resource, resource_info=attr_info,
                parent_resource=parent_resource, member_actions=member_actions,
                plugin=plugin, allow_pagination=pagination,
                allow_sorting=sorting, collection_actions=collection_actions)
            # new_controller.collection has replaced hyphens with underscores
            manager.NeutronManager.set_plugin_for_resource(
                new_controller.collection, plugin)
            if path_prefix:
                manager.NeutronManager.add_resource_for_path_prefix(
                    collection, path_prefix)
        else:
            new_controller = utils.ShimCollectionsController(
                collection, None, legacy_controller,
                collection_actions=collection_actions,
                member_actions=member_actions,
                action_status=ext_res.controller.action_status,
                collection_methods=ext_res.collection_methods)
        # 将 new_controller 以 resource_name:new_controller 的方式保存到 NeutronManager 实例属性
        manager.NeutronManager.set_controller_for_resource(
            collection_key, new_controller)
    policy.reset()

Core API 请求处理

根控制器 V2Controller 没有显式地声明 Core Controller，但其实所有 Core API Request 都在 method def _lookup(self, collection, *remainder): 中。所有未显式定义的 URL path 都被路由到 _lookup 方法中得到了处理。

在 initialize_all() 阶段已经把 Core Controllers 都注册到 NeutronManager 的实例属性 self.resource_controller_mappings 中了，这里再根据 API Request 的类型（e.g. networks、subnets）从 NeutronManager 的实例属性中取出。

Network Controller 的实例属性可以看出，其关联到了 Core Plugin ML2，并在该 Plugin 类中实现了对 network 这个 Resource 的 “视图函数”。比如：API 请求 GET /v2.0/networks 的视图函数是 Ml2Plugin.get_networks。实际上所有 Core Resources 都会关联到同一个 Core Plugin，但 Extension Resources 就会根据不同的类型关联到相应的 Service Plugins 上。

Neutron 就是通过这种方式实现了从 Neutron API 层到 Neutron Plugin 层的调用封装。

# /opt/stack/neutron/neutron/plugins/ml2/plugin.py

    @db_api.retry_if_session_inactive()
    def get_networks(self, context, filters=None, fields=None,
                     sorts=None, limit=None, marker=None, page_reverse=False):
        # NOTE(ihrachys) use writer manager to be able to update mtu
        # TODO(ihrachys) remove in Queens when mtu is not nullable
        with db_api.CONTEXT_WRITER.using(context):
            nets_db = super(Ml2Plugin, self)._get_networks(
                context, filters, None, sorts, limit, marker, page_reverse)

            # NOTE(ihrachys) pre Pike networks may have null mtus; update them
            # in database if needed
            # TODO(ihrachys) remove in Queens+ when mtu is not nullable
            net_data = []
            for net in nets_db:
                if net.mtu is None:
                    net.mtu = self._get_network_mtu(net, validate=False)
                net_data.append(self._make_network_dict(net, context=context))

            self.type_manager.extend_networks_dict_provider(context, net_data)
            nets = self._filter_nets_provider(context, net_data, filters)
        return [db_utils.resource_fields(net, fields) for net in nets]

Extension API 请求处理

Extensions API 的本质是一个 WSGI Middleware 而非 WSGI Application。

# /opt/stack/neutron/neutron/api/extensions.py
import routes


def plugin_aware_extension_middleware_factory(global_config, **local_config):
    """Paste factory."""
    def _factory(app):
        ext_mgr = PluginAwareExtensionManager.get_instance()
        # ExtensionMiddleware 是 Extensions middleware for WSGI（路由隐射、视图函数）的封装，接收 Extensions  Resources Request 并进行处理
        return ExtensionMiddleware(app, ext_mgr=ext_mgr)
    return _factory


class ExtensionMiddleware(base.ConfigurableMiddleware):
    """Extensions middleware for WSGI."""
    
    def __init__(self, application,
    ...
        # extended resources
        for resource in self.ext_mgr.get_resources():
        ...
            # 自定义 Actions
            for action, method in resource.collection_actions.items():
                conditions = dict(method=[method])
                path = "/%s/%s" % (resource.collection, action)
                with mapper.submapper(controller=resource.controller,
                                      action=action,
                                      path_prefix=path_prefix,
                                      conditions=conditions) as submap:
                    submap.connect(path_prefix + path, path)
                    submap.connect(path_prefix + path + "_format",
                                   "%s.:(format)" % path)

            # 自定义 Methods
            for action, method in resource.collection_methods.items():
                conditions = dict(method=[method])
                path = "/%s" % resource.collection
                with mapper.submapper(controller=resource.controller,
                                      action=action,
                                      path_prefix=path_prefix,
                                      conditions=conditions) as submap:
                    submap.connect(path_prefix + path, path)
                    submap.connect(path_prefix + path + "_format",
                                   "%s.:(format)" % path)
                                   
            # 将 ResourceCollection、ResourceController、ResourceMemberAction 映射起来
            mapper.resource(resource.collection, resource.collection,
                            controller=resource.controller,
                            member=resource.member_actions,
                            parent_resource=resource.parent,
                            path_prefix=path_prefix)
...

从上述代码可以看出，虽然 Core API 使用了 Pecan 框架，但 Extension API 依旧使用了 routes 来进行 Mapper 的维护。

Neutron Agents

Neutron 具有大量的 Networking Agents 服务进程被分布式的部署到了各类节点之上运行，为 Neutron 提供各类网元功能的管理和执行服务。

Neutron Agent 的软件架构可以分为 3 层：

1. 北向接口层：提供 RPC 接口，供 Neutron Server 调用；

2. 数据模型转换层：将 RPC 数据模型转换为底层网元的 CLI 模型。

3. CLI 协议层：南向通过 CLI 协议栈对相应的底层网元进行配置；

例如，当 Neutron 为 VM 创建并绑定一个 Port 时，Linux Bridge Agent 和 OvS Agent 就会执行一系列指令。

OvS Agents

Neutron Agent 的程序入口依旧定义在 setup.cfg 文件中，我们这里主要以 OvS Agent 为例关注 neutron-openvswitch-agent.service 服务进程的启动流程。

# /opt/stack/neutron/setup.cfg
neutron-openvswitch-agent = neutron.cmd.eventlet.plugins.ovs_neutron_agent:main

# /opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/main.py
_main_modules = {
    'ovs-ofctl': 'neutron.plugins.ml2.drivers.openvswitch.agent.openflow.'
                 'ovs_ofctl.main',
    'native': 'neutron.plugins.ml2.drivers.openvswitch.agent.openflow.'
                 'native.main',
}

def main():
    common_config.init(sys.argv[1:])
    driver_name = cfg.CONF.OVS.of_interface
    mod_name = _main_modules[driver_name]
    mod = importutils.import_module(mod_name)
    mod.init_config()
    common_config.setup_logging()
    profiler.setup("neutron-ovs-agent", cfg.CONF.host)
    mod.main()

这里可以看见 OvS Agent 有两种不同的启动模式 ovs-ofctl 和 native，通过配置项 of_interface 来指定。通常为 ovs-ofctl，表示使用 Open vSwitch 的 ovs-ofctl 指令来操作流表。

通过RPC来连接Plugins与Agents

Plugins 和 Agents 之间通过 RPC/MQ 进行通信，两者都既是 RPC Producer（生产者）也是 RPC Consumer（消费者）。

Plugin RPC Consumer

Plugin 要想成为 Consumer，就需要向 RPC Server 申请，称为 Registered Endpoints。完成后，Plugin 就注册好了与对应的 Agent 进行 RPC 通信的 Endpoint（调用接口）了。

# /opt/stack/neutron/neutron/plugins/ml2/plugin.py

# 以 Core Plugin 的 RPC Listeners 启动方法为例
class Ml2Plugin(...):
...
    @log_helpers.log_method_call
    def start_rpc_listeners(self):
        """Start the RPC loop to let the plugin communicate with agents."""
        # 设置 ML2Plugin 与 Agents 通信的 endpoints
        self._setup_rpc()
        self.topic = topics.PLUGIN
        self.conn = n_rpc.Connection()
        # 将 endpoints 注册到 RPC Consumer 并创建 RPC Consumer 实例对象
        self.conn.create_consumer(self.topic, self.endpoints, fanout=False)
        self.conn.create_consumer(
            topics.SERVER_RESOURCE_VERSIONS,
            [resources_rpc.ResourcesPushToServerRpcCallback()],
            fanout=True)
        # process state reports despite dedicated rpc workers
        self.conn.create_consumer(topics.REPORTS,
                                  [agents_db.AgentExtRpcCallback()],
                                  fanout=False)
        # 以线程的方式启动 endpoint 中的 RPC servers 实例对象
        return self.conn.consume_in_threads()

    def start_rpc_state_reports_listener(self):
        self.conn_reports = n_rpc.Connection()
        self.conn_reports.create_consumer(topics.REPORTS,
                                          [agents_db.AgentExtRpcCallback()],
                                          fanout=False)
        return self.conn_reports.consume_in_threads()

   def _setup_rpc(self):
        """Initialize components to support agent communication."""
        # Agents endpoints 清单
        self.endpoints = [
            rpc.RpcCallbacks(self.notifier, self.type_manager),
            securitygroups_rpc.SecurityGroupServerRpcCallback(),
            dvr_rpc.DVRServerRpcCallback(),
            dhcp_rpc.DhcpRpcCallback(),
            agents_db.AgentExtRpcCallback(),
            metadata_rpc.MetadataRpcCallback(),
            resources_rpc.ResourcesPullRpcCallback()
        ]

其中最重要的 start_rpc_listeners 和 start_rpc_state_reports_listener 函数分别在上提到的 RpcWorker 和  RpcReportsWorker Workers 类被调用，从而实现 RPC Workers 的加载和运行。

OvS Agent RPC Consumer

在启动 ovs-agnet.service 服务进程过程中实例化 OVSNeutronAgent 类时，要求先完成 OvS Agent RPC Consumer 的配置。

class OVSNeutronAgent(...):
    ...
    def __init__(self, bridge_classes, ext_manager, conf=None):
    ...
        # 创建 RPC Consumer
        self.setup_rpc()
        ...

    def setup_rpc(self):
        self.plugin_rpc = OVSPluginApi(topics.PLUGIN)
        # allow us to receive port_update/delete callbacks from the cache
        self.plugin_rpc.register_legacy_notification_callbacks(self)
        self.sg_plugin_rpc = sg_rpc.SecurityGroupServerAPIShim(
            self.plugin_rpc.remote_resource_cache)
        self.dvr_plugin_rpc = dvr_rpc.DVRServerRpcApi(topics.PLUGIN)
        self.state_rpc = agent_rpc.PluginReportStateAPI(topics.REPORTS)

        # RPC network init
        self.context = context.get_admin_context_without_session()
        # Made a simple RPC call to Neutron Server.
        while True:
            try:
                self.state_rpc.has_alive_neutron_server(self.context)
            except oslo_messaging.MessagingTimeout as e:
                LOG.warning('l2-agent cannot contact neutron server. '
                            'Check connectivity to neutron server. '
                            'Retrying... '
                            'Detailed message: %(msg)s.', {'msg': e})
                continue
            break

        # 定义监听消费者函数类型
        # Define the listening consumers for the agent
        consumers = [[constants.TUNNEL, topics.UPDATE],
                     [constants.TUNNEL, topics.DELETE],
                     [topics.DVR, topics.UPDATE]]
        if self.l2_pop:
            consumers.append([topics.L2POPULATION, topics.UPDATE])
        self.connection = agent_rpc.create_consumers([self],
                                                     topics.AGENT,
                                                     consumers,
                                                     start_listening=False)

设置完 RPC Consumer 之后，OvS Agent 定义的 RPC Consumer 函数就能够从 MQ 接受并消费从 Plugin 发送过来的消息了。如下代码实现。

注意，OvS Agent 只会监听 UPDATE 和 DELETE 的 RPC 消息，并没有监听 CREATE，这是因为 Neutron Port 的创建并不由 OvS Agent 完成而是由 nova-compute.service 来完成。

    def port_update(self, context, **kwargs):
        port = kwargs.get('port')
        # Put the port identifier in the updated_ports set.
        # Even if full port details might be provided to this call,
        # they are not used since there is no guarantee the notifications
        # are processed in the same order as the relevant API requests
        self.updated_ports.add(port['id'])

    def port_delete(self, context, **kwargs):
        port_id = kwargs.get('port_id')
        self.deleted_ports.add(port_id)
        self.updated_ports.discard(port_id)

    def network_update(self, context, **kwargs):
        network_id = kwargs['network']['id']
        for port_id in self.network_ports[network_id]:
            # notifications could arrive out of order, if the port is deleted
            # we don't want to update it anymore
            if port_id not in self.deleted_ports:
                self.updated_ports.add(port_id)
        LOG.debug("network_update message processed for network "
                  "%(network_id)s, with ports: %(ports)s",
                  {'network_id': network_id,
                   'ports': self.network_ports[network_id]})

Plugin RPC 调用 OvS Agent 的示例

以 Create Port 业务流程中， Plugin RPC 调用 OvS Agent 的示例：

# /opt/stack/neutron/neutron/plugins/ml2/plugin.py

class Ml2Plugin(...):
...
    def create_port(self, context, port):
        ...
        return self._after_create_port(context, result, mech_context)

    def _after_create_port(self, context, result, mech_context):
        ...
        try:
            bound_context = self._bind_port_if_needed(mech_context)
        except ml2_exc.MechanismDriverError:
        ...
        return bound_context.current

    @db_api.retry_db_errors
    def _bind_port_if_needed(self, context, allow_notify=False,
                             need_notify=False, allow_commit=True):
        ...
            if not try_again:
                if allow_notify and need_notify:
                    self._notify_port_updated(context)
                return context
        ...
        return context

   def _notify_port_updated(self, mech_context):
        port = mech_context.current
        segment = mech_context.bottom_bound_segment
        if not segment:
            # REVISIT(rkukura): This should notify agent to unplug port
            network = mech_context.network.current
            LOG.debug("In _notify_port_updated(), no bound segment for "
                      "port %(port_id)s on network %(network_id)s",
                      {'port_id': port['id'], 'network_id': network['id']})
            return
        self.notifier.port_update(mech_context._plugin_context, port,
                                  segment[api.NETWORK_TYPE],
                                  segment[api.SEGMENTATION_ID],
                                  segment[api.PHYSICAL_NETWORK])


# /opt/stack/neutron/neutron/plugins/ml2/rpc.py
class AgentNotifierApi(...):
...
    def port_update(self, context, port, network_type, segmentation_id,
                    physical_network):
        # 构建 RPC Client
        cctxt = self.client.prepare(topic=self.topic_port_update,
                                    fanout=True)
        # 发送 RPC 消息
        cctxt.cast(context, 'port_update', port=port,
                   network_type=network_type, segmentation_id=segmentation_id,
                   physical_network=physical_network)

最终 Plugin 发出的 RPC 消息就会被订阅该 Target 的 Agent Sonsumer 接收到了。

比如被 OvS Agent 接收到该消息，并执行后续的动作。

# /opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    def port_update(self, context, **kwargs):
        port = kwargs.get('port')
        # Put the port identifier in the updated_ports set.
        # Even if full port details might be provided to this call,
        # they are not used since there is no guarantee the notifications
        # are processed in the same order as the relevant API requests
        self.updated_ports.add(port['id'])

Plugin Callback

除了上述举例直接调用 RPC 函数（call、cast）的方式之外，Plugins 还实现了一套 Callback System 机制。

Callback System 与 RPC 一样是为了实现通信，不同之处在于：

• RPC 是为了实现 neutron-server 与 agent 不同服务进程之间的任务消息传递；

• Callback System 是为了实现同一进程内部的、Core and Service components 之间的通信。

Callback System 用于传递 Resource 的 Lifecycle Events（e.g. before creation, before deletion, etc.），让不同的 Core 和 Services 之间、以及让不同的 Services 之间可以感知到特定 Resource 的状态变化。