MediaPipe框架- 常见问题

FAQ

常见问题

How to convert ImageFrames and GpuBuffers

如何转换ImageFrames和GpuBuffers

The Calculators ImageFrameToGpuBufferCalculator and GpuBufferToImageFrameCalculator convert back and forth between packets of type ImageFrame and GpuBuffer. ImageFrame refers to image data in CPU memory in any of a number of bitmap image formats. GpuBuffer refers to image data in GPU memory. You can find more detail in the Framework Concepts section GpuBuffer to ImageFrame Converters. You can see an example in:

​计算器ImageFrameToGpuBufferCalculator和GpuBufferToImageFrameCalculator在ImageFrame和GpuBuffer类型的数据包之间来回转换。ImageFrame指的是CPU内存中的任何位图图像格式的图像数据。GpuBuffer是指GPU内存中的图像数据。可以在框架概念部分GpuBuffer到ImageFrame转换器中找到更多详细信息。可以在中看到一个示例：

• object_detection_mobile_cpu.pbtxt

How to visualize perception results

如何将感知结果可视化

The AnnotationOverlayCalculator allows perception results, such as bounding boxes, arrows, and ovals, to be superimposed on the video frames aligned with the recognized objects. The results can be displayed in a diagnostic window when running on a workstation, or in a texture frame when running on device. You can see an example use of AnnotationOverlayCalculator in:

​AnnotationOverlayCalculator允许将感知结果（如边界框、箭头和椭圆）叠加在与识别对象对齐的视频帧上。在工作站上运行时，结果可以显示在诊断窗口中，在设备上运行时可以显示在纹理框中。可以在中看到AnnotationOverlayCalculator的使用示例：

• face_detection_mobile_gpu.pbtxt.

How to run calculators in parallel

如何并行运行计算器

Within a calculator graph, MediaPipe routinely runs separate calculator nodes in parallel. MediaPipe maintains a pool of threads, and runs each calculator as soon as a thread is available and all of its inputs are ready. Each calculator instance is only run for one set of inputs at a time, so most calculators need only to be thread-compatible and not thread-safe.

在计算器图中，MediaPipe通常并行运行单独的计算器节点。MediaPipe维护一个线程池，并在线程可用且所有输入就绪后立即运行每个计算器。每个计算器实例一次只为一组输入运行，因此大多数计算器只需要是线程兼容的，而不是线程安全的。

In order to enable one calculator to process multiple inputs in parallel, there are two possible approaches:

为了使一个计算器能够并行处理多个输入，有两种可能的方法：

1.Define multiple calculator nodes and dispatch input packets to all nodes.

1.定义多个计算器节点，并将输入数据包分派到所有节点。

2.Make the calculator thread-safe and configure its max_in_flight setting.

​2.使计算器线程安全，并配置其max_in_flight设置。

The first approach can be followed using the calculators designed to distribute packets across other calculators, such as RoundRobinDemuxCalculator. A single RoundRobinDemuxCalculator can distribute successive packets across several identically configured ScaleImageCalculator nodes.

​第一种方法可以使用设计用于在其他计算器（如RoundRobinMuxCalculator）之间分发数据包的计算器来实现。单个RoundRobinMuxCalculator可以将连续的数据包分布在几个配置相同的ScaleImageCalculator节点上。

The second approach allows up to max_in_flight invocations of the CalculatorBase::Process method on the same calculator node. The output packets from CalculatorBase::Process are automatically ordered by timestamp before they are passed along to downstream calculators.

​第二种方法允许在同一计算器节点上最多调用CalculatorBase::Process方法的max_in_flight。CalculatorBase::Process的输出数据包在传递给下游计算器之前会自动按时间戳排序。

With either aproach, you must be aware that the calculator running in parallel cannot maintain internal state in the same way as a normal sequential calculator.

对于任何一种方法，都必须意识到并行运行的计算器无法像正常的顺序计算器那样保持内部状态。

Output timestamps when using ImmediateInputStreamHandler

使用ImmediateInputStreamHandler时的输出时间戳

The ImmediateInputStreamHandler delivers each packet as soon as it arrives at an input stream. As a result, it can deliver a packet with a higher timestamp from one input stream before delivering a packet with a lower timestamp from a different input stream. If these input timestamps are both used for packets sent to one output stream, that output stream will complain that the timestamps are not monotonically increasing. In order to remedy this, the calculator must take care to output a packet only after processing is complete for its timestamp. This could be accomplished by waiting until input packets have been received from all inputstreams for that timestamp, or by ignoring a packet that arrives with a timestamp that has already been processed.

​一旦数据包到达输入流，ImmediateInputStreamHandler就会立即传递数据包。因此，它可以在从不同的输入流传递具有较低时间戳的分组之前，从一个输入流传递带有较高时间戳的数据包。如果这些输入时间戳都用于发送到一个输出流的分组，则该输出流将抱怨时间戳不是单调增加的。为了解决这一问题，计算器必须注意仅在处理完数据包的时间戳后才输出数据包。这可以通过等待直到已经从该时间戳的所有输入流接收到输入分组，或者通过忽略带有已经处理的时间戳的分组来实现。

How to change settings at runtime

如何在运行时更改设置

There are two main approaches to changing the settings of a calculator graph while the application is running:

有两种主要方法可以在应用程序运行时更改计算器图的设置：

1.Restart the calculator graph with modified CalculatorGraphConfig.

​1.使用修改后的CalculatorGraphConfig重新启动计算器图。

2.Send new calculator options through packets on graph input-streams.

2.通过图输入流上的数据包发送新的计算器选项。

The first approach has the advantage of leveraging CalculatorGraphConfig processing tools such as "subgraphs". The second approach has the advantage of allowing active calculators and packets to remain in-flight while settings change. Mediapipe contributors are currently investigating alternative approaches to achieve both of these advantages.

​第一种方法的优点是利用CalculatorGraphConfig处理工具，如“子图”。第二种方法的优点是允许活动计算器和数据包在设置更改时保持在飞行中。Mediapipe贡献者目前正在研究实现这两个优势的替代方法。

How to process realtime input streams

如何处理实时输入流

The mediapipe framework can be used to process data streams either online or offline. For offline processing, packets are pushed into the graph as soon as calculators are ready to process those packets. For online processing, one packet for each frame is pushed into the graph as that frame is recorded.

mediapipe框架可用于在线或离线处理数据流。对于离线处理，一旦计算器准备好处理这些数据包，数据包就会被推入图中。对于在线处理，每个帧的一个数据包在记录该帧时被推入到图中。

The MediaPipe framework requires only that successive packets be assigned monotonically increasing timestamps. By convention, realtime calculators and graphs use the recording time or the presentation time as the timestamp for each packet, with each timestamp representing microseconds since Jan/1/1970:00:00:00. This allows packets from various sources to be processed in a globally consistent order.

MediaPipe框架只要求为连续的数据包分配单调增加的时间戳。按照惯例，实时计算器和图使用记录时间或显示时间作为每个数据包的时间戳，每个时间戳表示自1970年1月1日0:00:00以来的微秒。这允许以全局一致的顺序处理来自各种源的数据包。

Normally for offline processing, every input packet is processed and processing continues as long as necessary. For online processing, it is often necessary to drop input packets in order to keep pace with the arrival of input data frames. When inputs arrive too frequently, the recommended technique for dropping packets is to use the MediaPipe calculators designed specifically for this purpose such as FlowLimiterCalculator and PacketClonerCalculator.

​通常，对于离线处理，每个输入数据包都会被处理，并且处理会持续到必要的时间。对于在线处理，为了跟上输入数据帧的到来，通常需要丢弃输入分组。当输入到达频率过高时，建议丢弃数据包的技术是使用专门为此目的设计的MediaPipe计算器，如FlowLimiterCalculator和PacketClonerCalculator。

For online processing, it is also necessary to promptly determine when processing can proceed. MediaPipe supports this by propagating timestamp bounds between calculators. Timestamp bounds indicate timestamp intervals that will contain no input packets, and they allow calculators to begin processing for those timestamps immediately. Calculators designed for realtime processing should carefully calculate timestamp bounds in order to begin processing as promptly as possible. For example, the MakePairCalculator uses the SetOffset API to propagate timestamp bounds from input streams to output streams.

​对于在线处理，还需要及时确定何时可以进行处理。MediaPipe通过在计算器之间传播时间戳界限来支持这一点。时间戳界限指示不包含输入数据包的时间戳间隔，并且它们允许计算器立即开始处理这些时间戳。为实时处理设计的计算器应该仔细计算时间戳界限，以便尽快开始处理。例如，MakePairCalculator使用SetOffset API将时间戳界限从输入流传播到输出流。

Can I run MediaPipe on MS Windows?

我可以在MS Windows上运行MediaPipe吗？

Currently MediaPipe portability supports Debian Linux, Ubuntu Linux, MacOS, Android, and iOS. The core of MediaPipe framework is a C++ library conforming to the C++11 standard, so it is relatively easy to port to additional platforms.

目前，MediaPipe可移植性支持Debian Linux、Ubuntu Linux、MacOS、Android和iOS。MediaPipe框架的核心是一个符合C++11标准的C++库，因此相对容易移植到其他平台。