Nvidia Deepstream极致细节：2. Deepstream Python Meta数据解读

Nvidia Deepstream极致细节：2. Deepstream Python Meta数据解读

这一章节中，我们将介绍如何使用Nvidia Deepstream内自定义的Metadata，包括NvDsBatchMeta，NvDsFrameMeta，NvDsObjectMeta，NvDsClassifierMeta，NvDsDisplayMeta。

---

---

Gst-nvstreammux的输出是Gst Buffer以及NvDsBatchMeta。参见下图：

Gst Buffer是GStreamer中数据传输的基本单位。每个 Gst Buffer 都有关联的Meta数据。 DeepStream SDK 附加了 DeepStream Meta数据对象“NvDsBatchMeta”，在以下部分中进行了描述。NVIDIA DeepStream SDK Python API以及NVIDIA DeepStream SDK API Reference Documentation包含了详细描述。

1. 一个例子：deepstream_test_1

我们将从下面这个例子中，来详细理解和解读Meta数据的使用。

def osd_sink_pad_buffer_probe(pad,info,u_data):
    frame_number=0
    #Intiallizing object counter with 0.
    obj_counter = {
        PGIE_CLASS_ID_VEHICLE:0,
        PGIE_CLASS_ID_PERSON:0,
        PGIE_CLASS_ID_BICYCLE:0,
        PGIE_CLASS_ID_ROADSIGN:0
    }
    num_rects=0

    gst_buffer = info.get_buffer()
    if not gst_buffer:
        print("Unable to get GstBuffer ")
        return

    # Retrieve batch metadata from the gst_buffer
    # Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
    # C address of gst_buffer as input, which is obtained with hash(gst_buffer)
    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
    l_frame = batch_meta.frame_meta_list
    while l_frame is not None:
        try:
            # Note that l_frame.data needs a cast to pyds.NvDsFrameMeta
            # The casting is done by pyds.glist_get_nvds_frame_meta()
            # The casting also keeps ownership of the underlying memory
            # in the C code, so the Python garbage collector will leave
            # it alone.
            #frame_meta = pyds.glist_get_nvds_frame_meta(l_frame.data)
            frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
        except StopIteration:
            break

        frame_number=frame_meta.frame_num
        num_rects = frame_meta.num_obj_meta
        l_obj=frame_meta.obj_meta_list
        while l_obj is not None:
            try:
                # Casting l_obj.data to pyds.NvDsObjectMeta
                #obj_meta=pyds.glist_get_nvds_object_meta(l_obj.data)
                obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
            except StopIteration:
                break
            obj_counter[obj_meta.class_id] += 1
            obj_meta.rect_params.border_color.set(0.0, 0.0, 1.0, 0.0)
            try: 
                l_obj=l_obj.next
            except StopIteration:
                break

        # Acquiring a display meta object. The memory ownership remains in
        # the C code so downstream plugins can still access it. Otherwise
        # the garbage collector will claim it when this probe function exits.
        display_meta=pyds.nvds_acquire_display_meta_from_pool(batch_meta)
        display_meta.num_labels = 1
        py_nvosd_text_params = display_meta.text_params[0]
        # Setting display text to be shown on screen
        # Note that the pyds module allocates a buffer for the string, and the
        # memory will not be claimed by the garbage collector.
        # Reading the display_text field here will return the C address of the
        # allocated string. Use pyds.get_string() to get the string content.
        py_nvosd_text_params.display_text = "Frame Number={} Number of Objects={} Vehicle_count={} Person_count={}".format(frame_number, num_rects, obj_counter[PGIE_CLASS_ID_VEHICLE], obj_counter[PGIE_CLASS_ID_PERSON])

        # Now set the offsets where the string should appear
        py_nvosd_text_params.x_offset = 10
        py_nvosd_text_params.y_offset = 12

        # Font , font-color and font-size
        py_nvosd_text_params.font_params.font_name = "Serif"
        py_nvosd_text_params.font_params.font_size = 10
        # set(red, green, blue, alpha); set to White
        py_nvosd_text_params.font_params.font_color.set(1.0, 1.0, 1.0, 1.0)

        # Text background color
        py_nvosd_text_params.set_bg_clr = 1
        # set(red, green, blue, alpha); set to Black
        py_nvosd_text_params.text_bg_clr.set(0.0, 0.0, 0.0, 1.0)
        # Using pyds.get_string() to get display_text as string
        print(pyds.get_string(py_nvosd_text_params.display_text))
        pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)
        try:
            l_frame=l_frame.next
        except StopIteration:
            break
    return Gst.PadProbeReturn.OK

2. NvDsBatchMeta：基本的Metadata结构

DeepStream对Meta数据使用可扩展的标准结构。基本Meta数据结构NvDsBatchMeta从批处理级Meta数据开始，在Gst-nvstreammux插件中创建。 辅助Meta数据结构包含框架、对象、分类器和标签数据。DeepStream还提供了一种在批处理（Batch）、帧或对象级别添加用户特定Meta数据的机制。

DeepStream通过附加NvDsBatchMeta结构并将GstNvDsMetaType.meta_type设置为Gst-nvstreammux插件中的NVDS_BATCH_GST_META，将Meta数据附加到Gst buffer。当您的应用程序处理Gst buffer时，它可以遍历附加的Meta数据以找到NVDS_BATCH_GST_META。

函数gst_buffer_get_nvds_batch_meta()从Gst Buffer中提取NvDsBatchMeta。 （请参阅 sources/include/gstnvdsmeta.h 中的声明。）有关更多详细信息，请参阅 NVIDIA DeepStream SDK API。

3. Meta-data Overview

下图显示了metadata NvDsBatchMeta、NvDsFrameMeta、NvDsObjectMeta、NvDsClassifierMeta、NvDsDisplayMeta的结构和里面的结构内容。 逻辑非常简单。 首先，Gst-nvstreammux插件生成NvDsBatchMeta并存储在Gst Buffer中。 NvDsBatchMeta 仅表示一个批次的单桢Meta数据。当然，我们可以从中提取一帧Meta数据。而这个帧数据被称为NvDsFrameMeta。

需要注意的一件事：上图中的结构参数已经有了不小的变化。 （我猜上面的参数名称在 Deepstream 3.0 中。当前版本中更改了一些名称）。 有关更新的名称，请阅读目录 .../sources/includes 下的 nvdsmeta.h。

4. Code: NvDsBatchMeta

/**
 * Holds information about a formed batch containing frames from different
 * sources.
 * NOTE: Both Video and Audio metadata uses the same NvDsBatchMeta type.
 * NOTE: Audio batch metadata is formed within nvinferaudio plugin
 * and will not be corresponding to any one buffer output from nvinferaudio.
 * The NvDsBatchMeta for audio is attached to the last input buffer
 * when the audio batch buffering reach configurable threshold
 * (audio frame length) and this is when inference output is available.
 */
typedef struct _NvDsBatchMeta {
  NvDsBaseMeta base_meta;
  /** Holds the maximum number of frames in the batch. */
  guint max_frames_in_batch;
  /** Holds the number of frames now in the batch. */
  guint num_frames_in_batch;
  /** Holds a pointer to a pool of pointers of type @ref NvDsFrameMeta,
   representing a pool of frame metas. */
  NvDsMetaPool *frame_meta_pool;
  /** Holds a pointer to a pool of pointers of type NvDsObjMeta,
   representing a pool of object metas. */
  NvDsMetaPool *obj_meta_pool;
  /** Holds a pointer to a pool of pointers of type @ref NvDsClassifierMeta,
   representing a pool of classifier metas. */
  NvDsMetaPool *classifier_meta_pool;
  /** Holds a pointer to a pool of pointers of type @ref NvDsDisplayMeta,
   representing a pool of display metas. */
  NvDsMetaPool *display_meta_pool;
  /** Holds a pointer to a pool of pointers of type @ref NvDsUserMeta,
   representing a pool of user metas. */
  NvDsMetaPool *user_meta_pool;
  /** Holds a pointer to a pool of pointers of type @ref NvDsLabelInfo,
   representing a pool of label metas. */
  NvDsMetaPool *label_info_meta_pool;
  /** Holds a pointer to a list of pointers of type NvDsFrameMeta
   or NvDsAudioFrameMeta (when the batch represent audio batch),
   representing frame metas used in the current batch.
   */
  NvDsFrameMetaList *frame_meta_list;
  /** Holds a pointer to a list of pointers of type NvDsUserMeta,
   representing user metas in the current batch. */
  NvDsUserMetaList *batch_user_meta_list;
  /** Holds a lock to be set before accessing metadata to avoid
   simultaneous update by multiple components. */
  GRecMutex meta_mutex;
  /** Holds an array of user-specific batch information. */
  gint64 misc_batch_info[MAX_USER_FIELDS];
  /** For internal use. */
  gint64 reserved[MAX_RESERVED_FIELDS];
} NvDsBatchMeta;

在案例代码中，我们只用到了l_frame = batch_meta.frame_meta_list。

5. Code: NvDsFrameMeta

/**
 * Holds metadata for a frame in a batch.
 */
typedef struct _NvDsFrameMeta {
  /** Holds the base metadata for the frame. */
  NvDsBaseMeta base_meta;
  /** Holds the pad or port index of the Gst-streammux plugin for the frame
   in the batch. */
  guint pad_index;
  /** Holds the location of the frame in the batch. The frame's
   @ref NvBufSurfaceParams are at index @a batch_id in the @a surfaceList
   array of @ref NvBufSurface. */
  guint batch_id;
  /** Holds the current frame number of the source. */
  gint frame_num;
  /** Holds the presentation timestamp (PTS) of the frame. */
  guint64 buf_pts;
  /** Holds the ntp timestamp. */
  guint64 ntp_timestamp;
  /** Holds the source IDof the frame in the batch, e.g. the camera ID.
   It need not be in sequential order. */
  guint source_id;
  /** Holds the number of surfaces in the frame, required in case of
   multiple surfaces in the frame. */
  gint num_surfaces_per_frame;
  /* Holds the width of the frame at input to Gst-streammux. */
  guint source_frame_width;
  /* Holds the height of the frame at input to Gst-streammux. */
  guint source_frame_height;
  /* Holds the surface type of the subframe, required in case of
   multiple surfaces in the frame. */
  guint surface_type;
  /* Holds the surface index of tje subframe, required in case of
   multiple surfaces in the frame. */
  guint surface_index;
  /** Holds the number of object meta elements attached to current frame. */
  guint num_obj_meta;
  /** Holds a Boolean indicating whether inference is performed on the frame. */
  gboolean bInferDone;
  /** Holds a pointer to a list of pointers of type @ref NvDsObjectMeta
   in use for the frame. */
  NvDsObjectMetaList *obj_meta_list;
  /** Holds a pointer to a list of pointers of type @ref NvDsDisplayMeta
   in use for the frame. */
  NvDisplayMetaList *display_meta_list;
  /** Holds a pointer to a list of pointers of type @ref NvDsUserMeta
   in use for the frame. */
  NvDsUserMetaList *frame_user_meta_list;
  /** Holds additional user-defined frame information. */
  gint64 misc_frame_info[MAX_USER_FIELDS];
  /* Holds the width of the frame at output of Gst-streammux. */
  guint pipeline_width;
  /* Holds the height of the frame at output of Gst-streammux. */
  guint pipeline_height;
  /** For internal use. */
  gint64 reserved[MAX_RESERVED_FIELDS];
} NvDsFrameMeta;

在案例代码中，我们使用了如下几个元素：

frame_number=frame_meta.frame_num
num_rects = frame_meta.num_obj_meta
l_obj=frame_meta.obj_meta_list

实际上，NvDsFrameMeta包含了一些其他有用的元素，比如

• pad_index：Holds the pad or port index of the Gst-streammux plugin for the frame in the batch. 如果我们有超过1路视频的时候，我们可以通过获得这个参数来判断这一桢数据是来自哪一路视频。当然，我们也可以用这个参数：source_id。对我来说，这两个参数应该是一样的，因为输入端的视频FPS是相同的，并且batch数量等于视频数量；
• source_frame_width：source_frame_height：指的是在Gst-streammux的输入端图像的尺寸。（streammux输入输出图像尺寸可能不同）；
• obj_meta_list, display_meta_list, frame_user_meta_list：所以说，NvDsFrameMeta是和这三个模块都打通的。

6. Code: NvDsObjectMeta

/**
 * Holds metadata for an object in the frame.
 */
typedef struct _NvDsObjectMeta {
  NvDsBaseMeta base_meta;
  /** Holds a pointer to the parent @ref NvDsObjectMeta. Set to NULL if
   no parent exists. */
  struct _NvDsObjectMeta *parent;
  /** Holds a unique component ID that identifies the metadata
   in this structure. */
  gint unique_component_id;
  /** Holds the index of the object class inferred by the primary
   detector/classifier. */
  gint class_id;
  /** Holds a unique ID for tracking the object. @ref UNTRACKED_OBJECT_ID
   indicates that the object has not been tracked. */
  guint64 object_id;
  /** Holds a structure containing bounding box parameters of the object when
    detected by detector. */
  NvDsComp_BboxInfo detector_bbox_info;
  /** Holds a structure containing bounding box coordinates of the object when
   * processed by tracker. */
  NvDsComp_BboxInfo tracker_bbox_info;
  /** Holds a confidence value for the object, set by the inference
   component. confidence will be set to -0.1, if "Group Rectangles" mode of
   clustering is chosen since the algorithm does not preserve confidence
   values. Also, for objects found by tracker and not inference component,
   confidence will be set to -0.1 */
  gfloat confidence;
  /** Holds a confidence value for the object set by nvdcf_tracker.
   * tracker_confidence will be set to -0.1 for KLT and IOU tracker */
  gfloat tracker_confidence;
  /** Holds a structure containing positional parameters of the object
   * processed by the last component that updates it in the pipeline.
   * e.g. If the tracker component is after the detector component in the
   * pipeline then positinal parameters are from tracker component.
   * Positional parameters are clipped so that they do not fall outside frame
   * boundary. Can also be used to overlay borders or semi-transparent boxes on
   * objects. @see NvOSD_RectParams. */
  NvOSD_RectParams rect_params;
  /** Holds mask parameters for the object. This mask is overlayed on object
   * @see NvOSD_MaskParams. */
  NvOSD_MaskParams mask_params;
  /** Holds text describing the object. This text can be overlayed on the
   standard text that identifies the object. @see NvOSD_TextParams. */
  NvOSD_TextParams text_params;
  /** Holds a string describing the class of the detected object. */
  gchar obj_label[MAX_LABEL_SIZE];
  /** Holds a pointer to a list of pointers of type @ref NvDsClassifierMeta. */
  NvDsClassifierMetaList *classifier_meta_list;
  /** Holds a pointer to a list of pointers of type @ref NvDsUserMeta. */
  NvDsUserMetaList *obj_user_meta_list;
  /** Holds additional user-defined object information. */
  gint64 misc_obj_info[MAX_USER_FIELDS];
  /** For internal use. */
  gint64 reserved[MAX_RESERVED_FIELDS];
}NvDsObjectMeta;

NvDsObjectMeta包含了每个检测到的object的信息，在示例代码中，我们只是用了一个元素：obj_meta.rect_params.border_color.set(0.0, 0.0, 1.0, 0.0)。但实际上，NvDsObjectMeta包含了一些其他有用的元素，比如

• class_id：Holds the index of the object class inferred by the primary detector/classifier。通过这个class_id，我们可以知道监测到的物体属于哪一类；
• object_id：Holds a unique ID for tracking the object。这个参数其实很有用，但我们需要计数的时候，有一些物体它会一直出现在屏幕中，只要这个物体可以被检测到，并且track上，那么这个id号就不会变。
• rect_params, mask_params, text_params：这三个分别对应识别到物体的框框，mask，以及文字。
• obj_user_meta_list，classifier_meta_list

7. Code: NvDsClassifierMeta

typedef struct _NvDsClassifierMeta {
  NvDsBaseMeta base_meta;
  /** Holds the number of outputs/labels produced by the classifier. */
  guint num_labels;
  /** Holds a unique component ID for the classifier metadata. */
  gint unique_component_id;
  /** Holds a pointer to a list of pointers of type @ref NvDsLabelInfo. */
  NvDsLabelInfoList *label_info_list;
} NvDsClassifierMeta;

8. Code: NvDsDisplayMeta

typedef struct NvDsDisplayMeta {
  NvDsBaseMeta base_meta;
  /** Holds the number of rectangles described. */
  guint num_rects;
  /** Holds the number of labels (strings) described. */
  guint num_labels;
  /** Holds the number of lines described. */
  guint num_lines;
  /** Holds the number of arrows described. */
  guint num_arrows;
  /** Holds the number of circles described. */
  guint num_circles;
  /** Holds an array of positional parameters for rectangles.
   Used to overlay borders or semi-transparent rectangles,
   as required by the application. @see NvOSD_RectParams. */
  NvOSD_RectParams rect_params[MAX_ELEMENTS_IN_DISPLAY_META];
  /** Holds an array of text parameters for user-defined strings that can be
   overlayed using this structure. @see NvOSD_TextParams. */
  NvOSD_TextParams text_params[MAX_ELEMENTS_IN_DISPLAY_META];
  /** Holds an array of line parameters that the user can use to draw polygons
   in the frame, e.g. to show a RoI in the frame. @see NvOSD_LineParams. */
  NvOSD_LineParams line_params[MAX_ELEMENTS_IN_DISPLAY_META];
  /** Holds an array of arrow parameters that the user can use to draw arrows
   in the frame. @see NvOSD_ArrowParams */
  NvOSD_ArrowParams arrow_params[MAX_ELEMENTS_IN_DISPLAY_META];
  /** Holds an array of circle parameters that the user can use to draw circles
   in the frame. @see NvOSD_CircleParams */
  NvOSD_CircleParams circle_params[MAX_ELEMENTS_IN_DISPLAY_META];
  /** Holds an array of user-defined OSD metadata. */
  gint64 misc_osd_data[MAX_USER_FIELDS];
  /** For internal use. */
  gint64 reserved[MAX_RESERVED_FIELDS];
} NvDsDisplayMeta;