MediaPipe框架- 图

Graphs

图

Graph

图

A CalculatorGraphConfig proto specifies the topology and functionality of a MediaPipe graph. Each node in the graph represents a particular calculator or subgraph, and specifies necessary configurations, such as registered calculator/subgraph type, inputs, outputs and optional fields, such as node-specific options, input policy and executor, discussed in Synchronization.

​CalculatorGraphConfig原型指定MediaPipe图形的拓扑结构和功能。图中的每个节点表示一个特定的计算器或子图，并指定必要的配置，如注册的计算器/子图类型、输入、输出和可选字段，如同步中讨论的特定于节点的选项、输入策略和执行器。

CalculatorGraphConfig has several other fields to configure global graph-level settings, e.g. graph executor configs, number of threads, and maximum queue size of input streams. Several graph-level settings are useful for tuning the performance of the graph on different platforms (e.g., desktop v.s. mobile). For instance, on mobile, attaching a heavy model-inference calculator to a separate executor can improve the performance of a real-time application since this enables thread locality.

CalculatorGraphConfig还有几个其他字段用于配置全局图形级别设置，例如图形执行器配置、线程数和输入流的最大队列大小。几个图形级别设置可用于调整不同平台（例如，台式机与移动设备）上的图形性能。例如，在移动设备上，将沉重的模型推理计算器附加到单独的执行器可以提高实时应用程序的性能，因为这可以实现线程局部性。

Below is a trivial CalculatorGraphConfig example where we have series of passthrough calculators :

下面是一个琐碎的CalculatorGraphConfig示例，其中我们有一系列直通计算器：

# This graph named main_pass_throughcals_nosubgraph.pbtxt contains 4
# passthrough calculators.
input_stream: "in"
output_stream: "out"
node {
    calculator: "PassThroughCalculator"
    input_stream: "in"
    output_stream: "out1"
}
node {
    calculator: "PassThroughCalculator"
    input_stream: "out1"
    output_stream: "out2"
}
node {
    calculator: "PassThroughCalculator"
    input_stream: "out2"
    output_stream: "out3"
}
node {
    calculator: "PassThroughCalculator"
    input_stream: "out3"
    output_stream: "out"
}

MediaPipe offers an alternative C++ representation for complex graphs (e.g. ML pipelines, handling model metadata, optional nodes, etc.). The above graph may look like:

MediaPipe为复杂的图（例如ML管道、处理模型元数据、可选节点等）提供了一种替代的C++表示。上图可能看起来像：

CalculatorGraphConfig BuildGraphConfig() {
  Graph graph;

  // Graph inputs
  Stream<AnyType> in = graph.In(0).SetName("in");

  auto pass_through_fn = [](Stream<AnyType> in,
                            Graph& graph) -> Stream<AnyType> {
    auto& node = graph.AddNode("PassThroughCalculator");
    in.ConnectTo(node.In(0));
    return node.Out(0);
  };

  Stream<AnyType> out1 = pass_through_fn(in, graph);
  Stream<AnyType> out2 = pass_through_fn(out1, graph);
  Stream<AnyType> out3 = pass_through_fn(out2, graph);
  Stream<AnyType> out4 = pass_through_fn(out3, graph);

  // Graph outputs
  out4.SetName("out").ConnectTo(graph.Out(0));

  return graph.GetConfig();
}

See more details in Building Graphs in C++.

​请参阅C++中的构建图中的更多详细信息。

Subgraph

子图

To modularize a CalculatorGraphConfig into sub-modules and assist with re-use of perception solutions, a MediaPipe graph can be defined as a Subgraph. The public interface of a subgraph consists of a set of input and output streams similar to a calculator's public interface. The subgraph can then be included in a CalculatorGraphConfig as if it were a calculator. When a MediaPipe graph is loaded from a CalculatorGraphConfig, each subgraph node is replaced by the corresponding graph of calculators. As a result, the semantics and performance of the subgraph is identical to the corresponding graph of calculators.

为了将CalculatorGraphConfig模块化为子模块并帮助重用感知解决方案，可以将MediaPipe图定义为子图。子图的公共接口由一组输入和输出流组成，类似于计算器的公共接口。然后，子图可以被包括在CalculatorGraphConfig中，就像它是一个计算器一样。从CalculatorGraphConfig加载MediaPipe图时，每个子图节点都会被相应的计算器图替换。因此，子图的语义和性能与计算器的相应图相同。

Below is an example of how to create a subgraph named TwoPassThroughSubgraph.

下面是一个如何创建名为TwoPassThroughSubgraph的子图的示例。

1.Defining the subgraph.

1.定义子图。

# This subgraph is defined in two_pass_through_subgraph.pbtxt
# and is registered as "TwoPassThroughSubgraph"

type: "TwoPassThroughSubgraph"
input_stream: "out1"
output_stream: "out3"

node {
    calculator: "PassThroughCalculator"
    input_stream: "out1"
    output_stream: "out2"
}
node {
    calculator: "PassThroughCalculator"
    input_stream: "out2"
    output_stream: "out3"
}

The public interface to the subgraph consists of:

子图的公共接口包括：

• Graph input streams
• Graph output streams
• Graph input side packets
• Graph output side packets

2.Register the subgraph using BUILD rule mediapipe_simple_subgraph. The parameter register_as defines the component name for the new subgraph.

2.使用BUILD规则mediapipe_sample_subgraph注册子图。参数register_as定义了新子图的组件名称。

# Small section of BUILD file for registering the "TwoPassThroughSubgraph"
# subgraph for use by main graph main_pass_throughcals.pbtxt

mediapipe_simple_subgraph(
    name = "twopassthrough_subgraph",
    graph = "twopassthrough_subgraph.pbtxt",
    register_as = "TwoPassThroughSubgraph",
    deps = [
            "//mediapipe/calculators/core:pass_through_calculator",
            "//mediapipe/framework:calculator_graph",
    ],
)

3.Use the subgraph in the main graph.

3.使用主图中的子图。

# This main graph is defined in main_pass_throughcals.pbtxt
# using subgraph called "TwoPassThroughSubgraph"

input_stream: "in"
node {
    calculator: "PassThroughCalculator"
    input_stream: "in"
    output_stream: "out1"
}
node {
    calculator: "TwoPassThroughSubgraph"
    input_stream: "out1"
    output_stream: "out3"
}
node {
    calculator: "PassThroughCalculator"
    input_stream: "out3"
    output_stream: "out4"
}

Graph Options

图选项

It is possible to specify a "graph options" protobuf for a MediaPipe graph similar to the Calculator Options protobuf specified for a MediaPipe calculator. These "graph options" can be specified where a graph is invoked, and used to populate calculator options and subgraph options within the graph.

​可以为MediaPipe图指定一个“图选项”原型，类似于为MediaPipe计算器指定的计算器选项原型。这些“图选项”可以在调用图的位置指定，并用于填充图形中的计算器选项和子图选项。

In a CalculatorGraphConfig, graph options can be specified for a subgraph exactly like calculator options, as shown below:

在CalculatorGraphConfig中，可以为子图指定与计算器选项完全相同的图选项，如下所示：

node {
  calculator: "FlowLimiterCalculator"
  input_stream: "image"
  output_stream: "throttled_image"
  node_options: {
    [type.googleapis.com/mediapipe.FlowLimiterCalculatorOptions] {
      max_in_flight: 1
    }
  }
}

node {
  calculator: "FaceDetectionSubgraph"
  input_stream: "IMAGE:throttled_image"
  node_options: {
    [type.googleapis.com/mediapipe.FaceDetectionOptions] {
      tensor_width: 192
      tensor_height: 192
    }
  }
}

In a CalculatorGraphConfig, graph options can be accepted and used to populate calculator options, as shown below:

在CalculatorGraphConfig中，图选项可以被接受并用于填充计算器选项，如下所示：

graph_options: {
  [type.googleapis.com/mediapipe.FaceDetectionOptions] {}
}

node: {
  calculator: "ImageToTensorCalculator"
  input_stream: "IMAGE:image"
  node_options: {
    [type.googleapis.com/mediapipe.ImageToTensorCalculatorOptions] {
        keep_aspect_ratio: true
        border_mode: BORDER_ZERO
    }
  }
  option_value: "output_tensor_width:options/tensor_width"
  option_value: "output_tensor_height:options/tensor_height"
}

node {
  calculator: "InferenceCalculator"
  node_options: {
    [type.googleapis.com/mediapipe.InferenceCalculatorOptions] {}
  }
  option_value: "delegate:options/delegate"
  option_value: "model_path:options/model_path"
}

In this example, the FaceDetectionSubgraph accepts graph option protobuf FaceDetectionOptions. The FaceDetectionOptions is used to define some field values in the calculator options ImageToTensorCalculatorOptions and some field values in the subgraph options InferenceCalculatorOptions. The field values are defined using the option_value: syntax.

在本例中，FaceDetectionSubgraph接受FaceDetectionOptions的图选项protobuf。FaceDetectionOptions用于定义计算器选项ImageToTensorCalculatorOptions中的一些字段值和子图选项InferenceCalculatorOption中的一些域值。字段值是使用option_value:语法定义的。

In the CalculatorGraphConfig::Node protobuf, the fields node_options: and option_value: together define the option values for a calculator such as ImageToTensorCalculator. The node_options: field defines a set of literal constant values using the text protobuf syntax. Each option_value: field defines the value for one protobuf field using information from the enclosing graph, specifically from field values of the graph options of the enclosing graph. In the example above, the option_value: "output_tensor_width:options/tensor_width" defines the field ImageToTensorCalculatorOptions.output_tensor_width using the value of FaceDetectionOptions.tensor_width.

在CalculatorGraphConfig::Node protobuf中，字段Node_options:和option_value:一起定义计算器（如ImageToTensorCalculator）的选项值。node_options:字段使用text protobuf语法定义了一组文字常量值。每个option_value:字段使用封闭图中的信息，特别是封闭图的图选项的字段值，定义一个protobuf字段的值。在上面的示例中，option_value:“output_tensor_width:options/tensor_width”使用FaceDetectionOptions.tensor_width的值定义字段ImageToTensorCalculatorOptions.output_tensor_width。

The syntax of option_value: is similar to the syntax of input_stream:. The syntax is option_value: "LHS:RHS". The LHS identifies a calculator option field and the RHS identifies a graph option field. More specifically, the LHS and RHS each consists of a series of protobuf field names identifying nested protobuf messages and fields separated by '/'. This is known as the "ProtoPath" syntax. Nested messages that are referenced in the LHS or RHS must already be defined in the enclosing protobuf in order to be traversed using option_value:.

option_value的语法与input_stream的语法相似。语法为option_value:“LHS:RHS”。LHS标识计算器选项字段，RHS标识图选项字段。更具体地说，LHS和RHS分别由一系列protobuf字段名组成，用于标识嵌套的protobuf消息和用“/”分隔的字段。这就是所谓的“ProtoPath”语法。在LHS或RHS中引用的嵌套消息必须已在封装的protobuf中定义，才能使用option_value:进行遍历。

Cycles

循环

By default, MediaPipe requires calculator graphs to be acyclic and treats cycles in a graph as errors. If a graph is intended to have cycles, the cycles need to be annotated in the graph config. This page describes how to do that.

默认情况下，MediaPipe要求计算器图形是非循环的，并将图形中的循环视为错误。如果图要有循环，则需要在图配置中对循环进行注释。本页介绍了如何做到这一点。

NOTE: The current approach is experimental and subject to change. We welcome your feedback.

注：目前的方法是实验性的，可能会发生变化。我们欢迎反馈。

Please use the CalculatorGraphTest.Cycle unit test in mediapipe/framework/calculator_graph_test.cc as sample code. Shown below is the cyclic graph in the test. The sum output of the adder is the sum of the integers generated by the integer source calculator.

请在mediapipe/framework/caculator_graph_test.cc中使用 CalculatorGraphTest.Cycle单元测试作为示例代码。下面显示的是测试中的循环图。加法器的和输出是整数源计算器生成的整数的和。

This simple graph illustrates all the issues in supporting cyclic graphs.

这个简单的图说明了支持循环图的所有问题。

Back Edge Annotation

后边缘注释

We require that an edge in each cycle be annotated as a back edge. This allows MediaPipe’s topological sort to work, after removing all the back edges.

我们要求将每个循环中的一条边注释为后边缘。这允许MediaPipe的拓扑排序在删除所有后边缘后工作。

There are usually multiple ways to select the back edges. Which edges are marked as back edges affects which nodes are considered as upstream and which nodes are considered as downstream, which in turn affects the priorities MediaPipe assigns to the nodes.

通常有多种方法可以选择后边缘。哪些边被标记为后边缘会影响哪些节点被视为上游节点，哪些节点被认为是下游节点，这反过来又会影响MediaPipe分配给节点的优先级。

For example, the CalculatorGraphTest.Cycle test marks the old_sum edge as a back edge, so the Delay node is considered as a downstream node of the adder node and is given a higher priority. Alternatively, we could mark the sum input to the delay node as the back edge, in which case the delay node would be considered as an upstream node of the adder node and is given a lower priority.

例如，CalculatorGraphTest.Cycle测试将old_sum边缘标记为后边缘，因此Delay节点被视为加法器节点的下游节点，并被赋予更高的优先级。或者，我们可以将输入到延迟节点的和标记为后边缘，在这种情况下，延迟节点将被视为加法器节点的上游节点，并被赋予较低的优先级。

Initial Packet

初始数据包

For the adder calculator to be runnable when the first integer from the integer source arrives, we need an initial packet, with value 0 and with the same timestamp, on the old_sum input stream to the adder. This initial packet should be output by the delay calculator in the Open() method.

为了使加法器计算器在来自整数源的第一个整数到达时能够运行，我们需要在加法器的old_sum输入流上有一个值为0且具有相同时间戳的初始数据包。这个初始数据包应该由Open（）方法中的延迟计算器输出。

Delay in a Loop

环路中的延迟

Each loop should incur a delay to align the previous sum output with the next integer input. This is also done by the delay node. So the delay node needs to know the following about the timestamps of the integer source calculator:

每个循环都应该产生一个延迟，以便将上一个和输出与下一个整数输入对齐。这也是由延迟节点完成的。因此，延迟节点需要了解以下关于整数源计算器的时间戳的信息：

• The timestamp of the first output.

• 第一个输出的时间戳。

• The timestamp delta between successive outputs.

• 连续输出之间的时间戳增量。

We plan to add an alternative scheduling policy that only cares about packet ordering and ignores packet timestamps, which will eliminate this inconvenience.

我们计划添加一种只关心数据包排序而忽略数据包时间戳的替代调度策略，这将消除这种不便。

Early Termination of a Calculator When One Input Stream is Done

完成一个输入流时计算器的提前终止

By default, MediaPipe calls the Close() method of a non-source calculator when all of its input streams are done. In the example graph, we want to stop the adder node as soon as the integer source is done. This is accomplished by configuring the adder node with an alternative input stream handler, EarlyCloseInputStreamHandler.

默认情况下，MediaPipe在完成所有输入流后调用非源计算器的Close（）方法。在示例图中，我们希望在完成整数源后立即停止加法器节点。这是通过使用替代输入流处理程序EarlyCloseInputStreamHandler配置加法器节点来实现的。

Relevant Source Code

相关源代码

Delay Calculator

延迟计算器

Note the code in Open() that outputs the initial packet and the code in Process() that adds a (unit) delay to input packets. As noted above, this delay node assumes that its output stream is used alongside an input stream with packet timestamps 0, 1, 2, 3, ...

请注意Open（）中输出初始数据包的代码和Process（）中为输入数据包添加（单位）延迟的代码。如上所述，该延迟节点假设其输出流与具有分组时间戳0、1、2、3、…的输入流一起使用。。。

class UnitDelayCalculator : public Calculator {
 public:
  static absl::Status FillExpectations(
      const CalculatorOptions& extendable_options, PacketTypeSet* inputs,
      PacketTypeSet* outputs, PacketTypeSet* input_side_packets) {
    inputs->Index(0)->Set<int>("An integer.");
    outputs->Index(0)->Set<int>("The input delayed by one time unit.");
    return absl::OkStatus();
  }

  absl::Status Open() final {
    Output()->Add(new int(0), Timestamp(0));
    return absl::OkStatus();
  }

  absl::Status Process() final {
    const Packet& packet = Input()->Value();
    Output()->AddPacket(packet.At(packet.Timestamp().NextAllowedInStream()));
    return absl::OkStatus();
  }
};

Graph Config

图配置

Note the back_edge annotation and the alternative input_stream_handler.

请注意back_edge注释和可选的input_stream_handler。

node {
  calculator: 'GlobalCountSourceCalculator'
  input_side_packet: 'global_counter'
  output_stream: 'integers'
}
node {
  calculator: 'IntAdderCalculator'
  input_stream: 'integers'
  input_stream: 'old_sum'
  input_stream_info: {
    tag_index: ':1'  # 'old_sum'
    back_edge: true
  }
  output_stream: 'sum'
  input_stream_handler {
    input_stream_handler: 'EarlyCloseInputStreamHandler'
  }
}
node {
  calculator: 'UnitDelayCalculator'
  input_stream: 'sum'
  output_stream: 'old_sum'
}