Robot Operating System——深度解析手动加载动态库的运行模式

在《Robot Operating System——初探可执行文件、动态库运行模式》一文中，我们粗浅介绍了一种通过动态库运行ROS2主体逻辑的方法。但是并没有深入讲解。这是因为我们需要先掌握本节的内容，才能更方便的探知和理解ROS2中对动态库的各种使用方法。

这次我们要使用的例子是composition/src/dlopen_composition.cpp。它会编译成一个可执行文件。在使用时传递给它动态库的路径，这样就可以加载这些动态库并将其中的各个Node构造出来。

代码分析

可执行文件

预处理

进入main函数后，我们会将用户传递进来的动态库地址放到libraries中。

#define DLOPEN_COMPOSITION_LOGGER_NAME "dlopen_composition"

int main(int argc, char * argv[])
{
  // Force flush of the stdout buffer.
  setvbuf(stdout, NULL, _IONBF, BUFSIZ);

  if (argc < 2) {
    fprintf(stderr, "Requires at least one argument to be passed with the library to load\n");
    return 1;
  }
  rclcpp::init(argc, argv);
  rclcpp::Logger logger = rclcpp::get_logger(DLOPEN_COMPOSITION_LOGGER_NAME);

  std::vector<std::string> libraries;
  for (int i = 1; i < argc; ++i) {
    libraries.push_back(argv[i]);
  }

然后我们构造一个单线程运行时环境，待后续创建的Node加入其中。

  rclcpp::executors::SingleThreadedExecutor exec;

在ROS2中，如果我们使用手动加载动态库的运行模式，就要使用到class_loader::ClassLoader类。一个class_loader::ClassLoader对象对应一个动态库文件。而一个文件中可能有多个Node，而这些Node的类实体都是未知的。这样我们在构造Node对象时要使用rclcpp_components::NodeInstanceWrapper来统一表达它们。

  rclcpp::NodeOptions options;
  std::vector<std::unique_ptr<class_loader::ClassLoader>> loaders;
  std::vector<rclcpp_components::NodeInstanceWrapper> node_wrappers;

动态库路径转class_loader::ClassLoader

对于保存到libraries中的每个动态库路径，我们都要加载它，并将其转换成一个class_loader::ClassLoader对象。

for (auto library : libraries) {
    RCLCPP_INFO(logger, "Load library %s", library.c_str());
    auto loader = std::make_unique<class_loader::ClassLoader>(library);

通过class_loader::ClassLoader获取Node类名

然后通过ClassLoader::getAvailableClasses解析出该动态库文件中所有Node的类名。

    auto classes = loader->getAvailableClasses<rclcpp_components::NodeFactory>();

创建Node类的工厂类

我们无法直接通过类名来创建类，需要先将其转换成其对应Node类的工厂类。

    for (auto clazz : classes) {
      RCLCPP_INFO(logger, "Instantiate class %s", clazz.c_str());
      auto node_factory = loader->createInstance<rclcpp_components::NodeFactory>(clazz);

通过工厂类创建rclcpp_components::NodeInstanceWrapper

由于每个Node类都不相同，但是我们要用一个类来表达，这样不同Node类的工厂类就会通过NodeFactoryTemplate<NodeClassName>::create_node_instance创建一个相同类型rclcpp_components::NodeInstanceWrapper的对象。

      auto wrapper = node_factory->create_node_instance(options);

收尾

后续我们将Node对象对应的接口指针添加到单线程运行器exec中。这样整个流程就运行起来了。

      auto node = wrapper.get_node_base_interface();
      node_wrappers.push_back(wrapper);
      exec.add_node(node);
    }
    loaders.push_back(std::move(loader));
  }

  exec.spin();

  for (auto wrapper : node_wrappers) {
    exec.remove_node(wrapper.get_node_base_interface());
  }
  node_wrappers.clear();

  rclcpp::shutdown();

  return 0;
}

编译

add_executable(dlopen_composition
  src/dlopen_composition.cpp)
ament_target_dependencies(dlopen_composition
  "class_loader"
  "rclcpp"
  "rclcpp_components")

可以看到编译过程没有链接任何和Node相关的动态库。

动态链接库

我们挑选composition/src/talker_component.cpp作为例子来讲解。

// Copyright 2016 Open Source Robotics Foundation, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

#include "composition/talker_component.hpp"

#include <chrono>
#include <iostream>
#include <memory>
#include <utility>

#include "rclcpp/rclcpp.hpp"
#include "std_msgs/msg/string.hpp"

using namespace std::chrono_literals;

namespace composition
{

// Create a Talker "component" that subclasses the generic rclcpp::Node base class.
// Components get built into shared libraries and as such do not write their own main functions.
// The process using the component's shared library will instantiate the class as a ROS node.
Talker::Talker(const rclcpp::NodeOptions & options)
: Node("talker", options), count_(0)
{
  // Create a publisher of "std_mgs/String" messages on the "chatter" topic.
  pub_ = create_publisher<std_msgs::msg::String>("chatter", 10);

  // Use a timer to schedule periodic message publishing.
  timer_ = create_wall_timer(1s, [this]() {return this->on_timer();});
}

void Talker::on_timer()
{
  auto msg = std::make_unique<std_msgs::msg::String>();
  msg->data = "Hello World: " + std::to_string(++count_);
  RCLCPP_INFO(this->get_logger(), "Publishing: '%s'", msg->data.c_str());
  std::flush(std::cout);

  // Put the message into a queue to be processed by the middleware.
  // This call is non-blocking.
  pub_->publish(std::move(msg));
}

}  // namespace composition

#include "rclcpp_components/register_node_macro.hpp"

// Register the component with class_loader.
// This acts as a sort of entry point, allowing the component to be discoverable when its library
// is being loaded into a running process.
RCLCPP_COMPONENTS_REGISTER_NODE(composition::Talker)

这段代码和本主题相关的部分是最后一行：RCLCPP_COMPONENTS_REGISTER_NODE。后面我们将分析RCLCPP_COMPONENTS_REGISTER_NODE和动态库加载class_loader::ClassLoader的关系。

原理分析

注册过程

我们追踪RCLCPP_COMPONENTS_REGISTER_NODE的实现

#define RCLCPP_COMPONENTS_REGISTER_NODE(NodeClass) \
  CLASS_LOADER_REGISTER_CLASS( \
    rclcpp_components::NodeFactoryTemplate<NodeClass>, \
    rclcpp_components::NodeFactory)

NodeFactoryTemplate是NodeFactory的子类。它封装了我们编写的Node类。同时提供了create_node_instance方法，用于返回一个用于统一不同Node类的转换器NodeInstanceWrapper对象。

/// NodeFactoryTemplate is a convenience class for instantiating components.
/**
 * The NodeFactoryTemplate class can be used to provide the NodeFactory interface for
 * components that implement a single-argument constructor and `get_node_base_interface`.
 */
template<typename NodeT>
class NodeFactoryTemplate : public NodeFactory
{
public:
  NodeFactoryTemplate() = default;
  virtual ~NodeFactoryTemplate() = default;

  /// Create an instance of a component
  /**
   * \param[in] options Additional options used in the construction of the component.
   */
  NodeInstanceWrapper
  create_node_instance(const rclcpp::NodeOptions & options) override
  {
    auto node = std::make_shared<NodeT>(options);

    return NodeInstanceWrapper(
      node, std::bind(&NodeT::get_node_base_interface, node));
  }
};

NodeInstanceWrapper底层通过泛化的std::shared_ptr<void> node_instance_来保存Node对象智能指针，又提供了get_node_base_interface方法用于返回Node对象的NodeBaseInterface智能指针。这样诸如SingleThreadedExecutor的Executor（执行器）就可以通过add方法将Node添加到运行调度环境。

/**
* @macro This is the macro which must be declared within the source (.cpp) file for each class that is to be exported as plugin.
* The macro utilizes a trick where a new struct is generated along with a declaration of static global variable of same type after it. The struct's constructor invokes a registration function with the plugin system. When the plugin system loads a library with registered classes in it, the initialization of static variables forces the invocation of the struct constructors, and all exported classes are automatically registerd.
*/
#define CLASS_LOADER_REGISTER_CLASS(Derived, Base) \
  CLASS_LOADER_REGISTER_CLASS_WITH_MESSAGE(Derived, Base, "")

/**
* @macro This macro is same as CLASS_LOADER_REGISTER_CLASS, but will spit out a message when the plugin is registered
* at library load time
*/
#define CLASS_LOADER_REGISTER_CLASS_WITH_MESSAGE(Derived, Base, Message) \
  CLASS_LOADER_REGISTER_CLASS_INTERNAL_HOP1_WITH_MESSAGE(Derived, Base, __COUNTER__, Message)

CLASS_LOADER_REGISTER_CLASS_INTERNAL_HOP1_WITH_MESSAGE中比较值得注意的是__COUNTER__的使用。
__COUNTER__是一个预处理器宏，它在每次被扩展时都会生成一个唯一的整数值。这个值从0开始，每次使用时递增1。它在需要生成唯一标识符的场景中非常有用。后面我们会分析为什么要用它参与宏的逻辑。

#define CLASS_LOADER_REGISTER_CLASS_INTERNAL_HOP1_WITH_MESSAGE(Derived, Base, UniqueID, Message) \
  CLASS_LOADER_REGISTER_CLASS_INTERNAL_WITH_MESSAGE(Derived, Base, UniqueID, Message)

#define CLASS_LOADER_REGISTER_CLASS_INTERNAL_WITH_MESSAGE(Derived, Base, UniqueID, Message) \
  namespace \
  { \
  struct ProxyExec ## UniqueID \
  { \
    typedef  Derived _derived; \
    typedef  Base _base; \
    ProxyExec ## UniqueID() \
    { \
      if (!std::string(Message).empty()) { \
        CONSOLE_BRIDGE_logInform("%s", Message);} \
      class_loader::impl::registerPlugin<_derived, _base>(#Derived, #Base); \
    } \
  }; \
  static ProxyExec ## UniqueID g_register_plugin_ ## UniqueID; \
  }  // namespace

CLASS_LOADER_REGISTER_CLASS_INTERNAL_WITH_MESSAGE是一个非常重要的宏，它是我们整个宏分析过程的核心。
它会使用ProxyExec连接__COUNTER__产生的自增数字，构成一个名字唯一的结构体名。比如第一次这个宏展开时，我们会得到ProxyExec0这样的结构体。
这个结构体的构造函数中，会调用class_loader::impl::registerPlugin<_derived, _base>来注册Node类的工厂类对象。

/**
 * @brief This function is called by the CLASS_LOADER_REGISTER_CLASS macro in plugin_register_macro.h to register factories.
 * Classes that use that macro will cause this function to be invoked when the library is loaded. The function will create a MetaObject (i.e. factory) for the corresponding Derived class and insert it into the appropriate FactoryMap in the global Base-to-FactoryMap map. Note that the passed class_name is the literal class name and not the mangled version.
 * @param Derived - parameteric type indicating concrete type of plugin
 * @param Base - parameteric type indicating base type of plugin
 * @param class_name - the literal name of the class being registered (NOT MANGLED)
 */
template<typename Derived, typename Base>
void registerPlugin(const std::string & class_name, const std::string & base_class_name)
{
  // Note: This function will be automatically invoked when a dlopen() call
  // opens a library. Normally it will happen within the scope of loadLibrary(),
  // but that may not be guaranteed.
  CONSOLE_BRIDGE_logDebug(
    "class_loader.impl: "
    "Registering plugin factory for class = %s, ClassLoader* = %p and library name %s.",
    class_name.c_str(), getCurrentlyActiveClassLoader(),
    getCurrentlyLoadingLibraryName().c_str());

  if (nullptr == getCurrentlyActiveClassLoader()) {
    CONSOLE_BRIDGE_logDebug("%s",
      "class_loader.impl: ALERT!!! "
      "A library containing plugins has been opened through a means other than through the "
      "class_loader or pluginlib package. "
      "This can happen if you build plugin libraries that contain more than just plugins "
      "(i.e. normal code your app links against). "
      "This inherently will trigger a dlopen() prior to main() and cause problems as class_loader "
      "is not aware of plugin factories that autoregister under the hood. "
      "The class_loader package can compensate, but you may run into namespace collision problems "
      "(e.g. if you have the same plugin class in two different libraries and you load them both "
      "at the same time). "
      "The biggest problem is that library can now no longer be safely unloaded as the "
      "ClassLoader does not know when non-plugin code is still in use. "
      "In fact, no ClassLoader instance in your application will be unable to unload any library "
      "once a non-pure one has been opened. "
      "Please refactor your code to isolate plugins into their own libraries.");
    hasANonPurePluginLibraryBeenOpened(true);
  }

  // Create factory
  impl::AbstractMetaObject<Base> * new_factory =
    new impl::MetaObject<Derived, Base>(class_name, base_class_name);
  new_factory->addOwningClassLoader(getCurrentlyActiveClassLoader());
  new_factory->setAssociatedLibraryPath(getCurrentlyLoadingLibraryName());


  // Add it to global factory map map
  getPluginBaseToFactoryMapMapMutex().lock();
  FactoryMap & factoryMap = getFactoryMapForBaseClass<Base>();
  if (factoryMap.find(class_name) != factoryMap.end()) {
    CONSOLE_BRIDGE_logWarn(
      "class_loader.impl: SEVERE WARNING!!! "
      "A namespace collision has occurred with plugin factory for class %s. "
      "New factory will OVERWRITE existing one. "
      "This situation occurs when libraries containing plugins are directly linked against an "
      "executable (the one running right now generating this message). "
      "Please separate plugins out into their own library or just don't link against the library "
      "and use either class_loader::ClassLoader/MultiLibraryClassLoader to open.",
      class_name.c_str());
  }
  factoryMap[class_name] = new_factory;
  getPluginBaseToFactoryMapMapMutex().unlock();

  CONSOLE_BRIDGE_logDebug(
    "class_loader.impl: "
    "Registration of %s complete (Metaobject Address = %p)",
    class_name.c_str(), reinterpret_cast<void *>(new_factory));
}

我们从工厂类对象创建开始分析这段代码

  impl::AbstractMetaObject<Base> * new_factory =
    new impl::MetaObject<Derived, Base>(class_name, base_class_name);

上面代码的 impl::AbstractMetaObject<Base>对象提供了create方法，用于创建rclcpp_components::NodeFactory。这个对应于可执行文件中，我们通过loader创建Node工厂类的过程。

// 可执行文件中的过程
      auto node_factory = loader->createInstance<rclcpp_components::NodeFactory>(clazz);

接下来的代码是告知工厂对象当前的环境。需要注意的是，我们分析的宏的逻辑，编译到动态链接库中，但是执行于进程加载该动态库之后。

这就牵扯到一个问题，有些代码是存在于其他文件（可执行文件或者其他动态库B）中，但是动态链接库（A）使用了它的符号。该动态库（A）在运行使用了符号的代码前，要确保加载它的进程已经把定义了符号对应的逻辑加载到内存中，这样才可以正确调用和运行。

下面getCurrentlyActiveClassLoader、getCurrentlyLoadingLibraryName就是逻辑在其他动态库中，但是在libtalker_component.so动态库中被调用的例子。

  new_factory->addOwningClassLoader(getCurrentlyActiveClassLoader());
  new_factory->setAssociatedLibraryPath(getCurrentlyLoadingLibraryName());

可以看到这两个符号在libtalker_component.so处于U状态：即该符号被使用但是没有实现。

它们实现于/opt/ros/jazzy/lib/libclass_loader.so。这个动态库会先于libtalker_component.so加载。

继续回到宏的代码分析上。

然后我们会将上述创建的Node工厂类对象保存到一个Map结构的静态变量中。

  // Add it to global factory map map
  getPluginBaseToFactoryMapMapMutex().lock();
  FactoryMap & factoryMap = getFactoryMapForBaseClass<Base>();
  if (factoryMap.find(class_name) != factoryMap.end()) {
    CONSOLE_BRIDGE_logWarn(
      "class_loader.impl: SEVERE WARNING!!! "
      "A namespace collision has occurred with plugin factory for class %s. "
      "New factory will OVERWRITE existing one. "
      "This situation occurs when libraries containing plugins are directly linked against an "
      "executable (the one running right now generating this message). "
      "Please separate plugins out into their own library or just don't link against the library "
      "and use either class_loader::ClassLoader/MultiLibraryClassLoader to open.",
      class_name.c_str());
  }
  factoryMap[class_name] = new_factory;
  getPluginBaseToFactoryMapMapMutex().unlock();

这个静态变量也存在于/opt/ros/jazzy/lib/libclass_loader.so中。它会随着libclass_loader.so被加载到进程中而被构建。

// src/class_loader_core.cpp
BaseToFactoryMapMap & getGlobalPluginBaseToFactoryMapMap()
{
  static BaseToFactoryMapMap instance;
  return instance;
}

FactoryMap & getFactoryMapForBaseClass(const std::string & typeid_base_class_name)
{
  BaseToFactoryMapMap & factoryMapMap = getGlobalPluginBaseToFactoryMapMap();
  std::string base_class_name = typeid_base_class_name;
  if (factoryMapMap.find(base_class_name) == factoryMapMap.end()) {
    factoryMapMap[base_class_name] = FactoryMap();
  }

  return factoryMapMap[base_class_name];
}

使用过程

第一步加载动态链接库文件，获取loader。这个过程涉及一系列加载动作，本文就不做分析了。具体可以参看github/ros2/class_loader/src/class_loader_core.cpp。

    auto loader = std::make_unique<class_loader::ClassLoader>(library);

获得loader后，我们要获取和它关联的所有Node类名。

    auto classes = loader->getAvailableClasses<rclcpp_components::NodeFactory>();

其底层实现如下

  template<class Base>
  std::vector<std::string> getAvailableClasses()
  {
    return class_loader::impl::getAvailableClasses<Base>(this);
  }

/**
 * @brief This function returns all the available class_loader in the plugin system that are derived from Base and within scope of the passed ClassLoader.
 * @param loader - The pointer to the ClassLoader whose scope we are within,
 * @return A vector of strings where each string is a plugin we can create
 */
template<typename Base>
std::vector<std::string> getAvailableClasses(ClassLoader * loader)
{
  boost::recursive_mutex::scoped_lock lock(getPluginBaseToFactoryMapMapMutex());

  FactoryMap & factory_map = getFactoryMapForBaseClass<Base>();
  std::vector<std::string> classes;
  std::vector<std::string> classes_with_no_owner;

  for (auto & it : factory_map) {
    AbstractMetaObjectBase * factory = it.second;
    if (factory->isOwnedBy(loader)) {
      classes.push_back(it.first);
    } else if (factory->isOwnedBy(nullptr)) {
      classes_with_no_owner.push_back(it.first);
    }
  }

  // Added classes not associated with a class loader (Which can happen through
  // an unexpected dlopen() to the library)
  classes.insert(classes.end(), classes_with_no_owner.begin(), classes_with_no_owner.end());
  return classes;
}

可以看到它会通过getFactoryMapForBaseClass获取动态库加载过程中注册到全局变量中的映射关系。然后遍历每个工厂类对象，查看它是否属于该loader。这个过程就和我们之前分析的new_factory->addOwningClassLoader(getCurrentlyActiveClassLoader());对应上了。

然后我们创建Node类的工厂类

      auto node_factory = loader->createInstance<rclcpp_components::NodeFactory>(clazz);

这个过程比较简单，代码如下

  /**
   * @brief  Generates an instance of loadable classes (i.e. class_loader).
   *
   * Same as createSharedInstance() except it returns a boost::shared_ptr.
   */
  template<class Base>
  boost::shared_ptr<Base> createInstance(const std::string & derived_class_name)
  {
    return boost::shared_ptr<Base>(
      createRawInstance<Base>(derived_class_name, true),
      boost::bind(&ClassLoader::onPluginDeletion<Base>, this, _1));
  }

接下来我们使用该工厂类创建Node类对象的转换器，从而拿到相关接口。
需要注意的是此处node_factory是rclcpp_components::NodeFactory类型的智能指针，而其实际指向的是rclcpp_components::NodeFactory的子类rclcpp_components::NodeFactoryTemplate<composition::Talker>对象。这就是通过父类的虚方法调用子类的实现，而子类知道具体的Node类类型，达到抽象的目的。

      auto wrapper = node_factory->create_node_instance(options);

  NodeInstanceWrapper
  create_node_instance(const rclcpp::NodeOptions & options) override
  {
    auto node = std::make_shared<NodeT>(options);

    return NodeInstanceWrapper(
      node, std::bind(&NodeT::get_node_base_interface, node));
  }

总结

1. RCLCPP_COMPONENTS_REGISTER_NODE宏会注册类的工厂类。
2. 注册的代码被编译到动态库静态变量的构造过程中。该过程会在动态库被进程加载时自动执行，从而达到自动在进程中注册的目的。
3. libclass_loader.so中有一个全局变量，会在进程加载该动态库时自动构建。之后Node工厂类的注册都会注册到该结构中。
4. 通过NodeInstanceWrapper封装不同Node类，进而达到抽象的目的。
5. Node类的构建发生在rclcpp_components::NodeFactoryTemplate<NodeClass>类中。NodeFactoryTemplate是NodeFactory的子类。在libclass_loader.so的全局变量中保存的是父类指针类型，后续使用时达到抽象目的。
6. 如果一个动态库中包含多个使用RCLCPP_COMPONENTS_REGISTER_NODE注册的Node类，则其底层的会通过__COUNTER__展开，按照顺序自动自增数字，保证每个静态变量名不一样。