deepstream Test4说明

原文:deepstream_python_apps/notebooks/deepstream_test_4.ipynb
代码比较难看，但是还好作者留下了NOTEBOOK

Deepstream Test4 description

TEST4建立在TEST1并添加展示了功能：
1、在pipeline中使用了 “nvmsgconv” 和 “nvmsgbroker” 插件
2、创建 NVDS_META_EVENT_MSG 元数据并将其加入缓存（buffer）.
3、对不同物体使用NVDS_META_EVENT_MSG
4、如果元数据通过“extMsg”字段扩展，提供了复制/释放功能。

对两个插件的说明太长了，直接机翻

“nvmsgconv”插件从缓冲区使用NVDS\u META\u EVENT\u MSG类型的元数据，并以Json格式生成“DeepStream Schema”负载。模式的静态属性以键值对的形式从配置文件中读取。检查dstest4\msgconv_配置.txt供参考。生成的负载作为NVDS\u META\u负载类型元数据附加到缓冲区。

“nvmsgbroker”插件从缓冲区提取NVDS\u META\u负载类型的元数据，并使用协议适配器api将该负载发送到服务器。
为不同类型的对象生成自定义元数据：除了NvDsEventMsgMeta结构中提供的公共字段外，用户还可以创建自定义对象并作为NVDS\u META\u EVENT\u MSG元数据附加到缓冲区。为此，NvDsEventMsgMeta提供了“extMsg”和“extMsgSize”字段。用户可以创建自定义结构，填充该结构，并将该结构的指针指定为“extMsg”，并相应地设置“extMsgSize”。如果自定义对象包含不能简单复制的字段，那么用户还应该提供复制和释放这些对象的功能。
附原文
“nvmsgconv” plugin uses NVDS_META_EVENT_MSG type of metadata from the buffer and generates the “DeepStream Schema” payload in Json format. Static properties of schema are read from configuration file in the form of key-value pair. Check dstest4_msgconv_config.txt for reference. Generated payload is attached as NVDS_META_PAYLOAD type metadata to the buffer.

“nvmsgbroker” plugin extracts NVDS_META_PAYLOAD type of metadata from the buffer and sends that payload to the server using protocol adaptor APIs.

Generating custom metadata for different type of objects: In addition to common fields provided in NvDsEventMsgMeta structure, user can also create custom objects and attach to buffer as NVDS_META_EVENT_MSG metadata. To do that NvDsEventMsgMeta provides “extMsg” and “extMsgSize” fields. User can create custom structure, fill that structure and assign the pointer of that structure as “extMsg” and set the “extMsgSize” accordingly. If custom object contains fields that can’t be simply mem copied then user should also provide function to copy and free those objects.

Installing necessary libraries (kafka)

In [ ]:

!sudo apt-get install libglib2.0 libglib2.0-dev
!sudo apt-get install libjansson4  libjansson-dev
!sudo apt-get install librdkafka1=0.11.3-1build1

Preparing the necessary arguments

cfg_file = Path to adaptor config file

input_file = Path to input x264 stream

proto_lib = Absolute path to adaptor library

conn_str = Connection string of backend server. Optional if it is part of config file.

topic = Name of message topic. Optional if it is part of connection string or config file.

no_display = To disable display. Default False schema_type = Type of minimal schema. 0= Full. 1= Minimal. Default =0

cfg_file = "cfg_kafka.txt" input_file = "" proto_lib = "/home/nvidia/Downloads/deepstream/sources/libs/kafka_protocol_adaptor/" conn_str = "host;port;topic" topic = "" no_display = False schema_type = 0

模块说明

Importing necessary libraries

import sys
sys.path.append('../')
sys.path.append('/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/')
import gi
gi.require_version('Gst', '1.0')
from gi.repository import GObject, Gst
from gi.repository import GLib
import sys
import platform
from optparse import OptionParser
from common.is_aarch_64 import is_aarch64
from common.bus_call import bus_call
from common.utils import long_to_int
import pyds

Declaring class label ids and other meta data requirements

MAX_DISPLAY_LEN=64
MAX_TIME_STAMP_LEN=32
PGIE_CLASS_ID_VEHICLE = 0
PGIE_CLASS_ID_BICYCLE = 1
PGIE_CLASS_ID_PERSON = 2
PGIE_CLASS_ID_ROADSIGN = 3
MUXER_OUTPUT_WIDTH=1920
MUXER_OUTPUT_HEIGHT=1080
MUXER_BATCH_TIMEOUT_USEC=4000000
input_file = "/home/nvidia/Downloads/deepstream/sources/python/apps/sample_720p.h264"
schema_type = 0
frame_number = 0
proto_lib = "/home/nvidia/Downloads/deepstream/sources/libs/kafka_protocol_adaptor/"
conn_str="localhost;2181;testTopic"
cfg_file = "cfg_kafka.txt"
topic = None
no_display = False

Necessary config files and meta data class labels

PGIE_CONFIG_FILE="dstest4_pgie_config.txt"
MSCONV_CONFIG_FILE="dstest4_msgconv_config.txt"


pgie_classes_str=["Vehicle", "TwoWheeler", "Person","Roadsign"]

Callback function for deep-copying an NvDsEventMsgMeta struct

def meta_copy_func(data,user_data):
    # Cast data to pyds.NvDsUserMeta
    user_meta=pyds.glist_get_nvds_user_meta(data)
    src_meta_data=user_meta.user_meta_data
    # Cast src_meta_data to pyds.NvDsEventMsgMeta
    srcmeta=pyds.glist_get_nvds_event_msg_meta(src_meta_data)
    # Duplicate the memory contents of srcmeta to dstmeta
    # First use pyds.get_ptr() to get the C address of srcmeta, then
    # use pyds.memdup() to allocate dstmeta and copy srcmeta into it.
    # pyds.memdup returns C address of the allocated duplicate.
    dstmeta_ptr=pyds.memdup(pyds.get_ptr(srcmeta), sys.getsizeof(pyds.NvDsEventMsgMeta))
    # Cast the duplicated memory to pyds.NvDsEventMsgMeta
    dstmeta=pyds.glist_get_nvds_event_msg_meta(dstmeta_ptr)

    # Duplicate contents of ts field. Note that reading srcmeat.ts
    # returns its C address. This allows to memory operations to be
    # performed on it.
    dstmeta.ts=pyds.memdup(srcmeta.ts, MAX_TIME_STAMP_LEN+1)

    # Copy the sensorStr. This field is a string property.
    # The getter (read) returns its C address. The setter (write)
    # takes string as input, allocates a string buffer and copies
    # the input string into it.
    # pyds.get_string() takes C address of a string and returns
    # the reference to a string object and the assignment inside the binder copies content.
    dstmeta.sensorStr=pyds.get_string(srcmeta.sensorStr)

    if(srcmeta.objSignature.size>0):
        dstmeta.objSignature.signature=pyds.memdup(srcmeta.objSignature.signature,srcMeta.objSignature.size)
        dstmeta.objSignature.size = srcmeta.objSignature.size;

    if(srcmeta.extMsgSize>0):
        if(srcmeta.objType==pyds.NvDsObjectType.NVDS_OBJECT_TYPE_VEHICLE):
            srcobj = pyds.glist_get_nvds_vehicle_object(srcmeta.extMsg);
            obj = pyds.alloc_nvds_vehicle_object();
            obj.type=pyds.get_string(srcobj.type)
            obj.make=pyds.get_string(srcobj.make)
            obj.model=pyds.get_string(srcobj.model)
            obj.color=pyds.get_string(srcobj.color)
            obj.license = pyds.get_string(srcobj.license)
            obj.region = pyds.get_string(srcobj.region)
            dstmeta.extMsg = obj;
            dstmeta.extMsgSize = sys.getsizeof(pyds.NvDsVehicleObject)
        if(srcmeta.objType==pyds.NvDsObjectType.NVDS_OBJECT_TYPE_PERSON):
            srcobj = pyds.glist_get_nvds_person_object(srcmeta.extMsg);
            obj = pyds.alloc_nvds_person_object()
            obj.age = srcobj.age
            obj.gender = pyds.get_string(srcobj.gender);
            obj.cap = pyds.get_string(srcobj.cap)
            obj.hair = pyds.get_string(srcobj.hair)
            obj.apparel = pyds.get_string(srcobj.apparel);
            dstmeta.extMsg = obj;
            dstmeta.extMsgSize = sys.getsizeof(pyds.NvDsVehicleObject);

    return dstmeta

Callback function for freeing an NvDsEventMsgMeta instance

def meta_free_func(data,user_data):
   user_meta=pyds.glist_get_nvds_user_meta(data)
   srcmeta=pyds.glist_get_nvds_event_msg_meta(user_meta.user_meta_data)

   # pyds.free_buffer takes C address of a buffer and frees the memory
   # It's a NOP if the address is NULL
   pyds.free_buffer(srcmeta.ts)
   pyds.free_buffer(srcmeta.sensorStr)

   if(srcmeta.objSignature.size > 0):
       pyds.free_buffer(srcmeta.objSignature.signature);
       srcmeta.objSignature.size = 0

   if(srcmeta.extMsgSize > 0):
       if(srcmeta.objType == pyds.NvDsObjectType.NVDS_OBJECT_TYPE_VEHICLE):
           obj =pyds.glist_get_nvds_vehicle_object(srcmeta.extMsg)
           pyds.free_buffer(obj.type);
           pyds.free_buffer(obj.color);
           pyds.free_buffer(obj.make);
           pyds.free_buffer(obj.model);
           pyds.free_buffer(obj.license);
           pyds.free_buffer(obj.region);
       if(srcmeta.objType == pyds.NvDsObjectType.NVDS_OBJECT_TYPE_PERSON):
           obj = pyds.glist_get_nvds_person_object(srcmeta.extMsg);
           pyds.free_buffer(obj.gender);
           pyds.free_buffer(obj.cap);
           pyds.free_buffer(obj.hair);
           pyds.free_buffer(obj.apparel);
       pyds.free_gbuffer(srcmeta.extMsg);
       srcmeta.extMsgSize = 0;

Generating meta data

def generate_vehicle_meta(data):
   obj = pyds.glist_get_nvds_vehicle_object(data);
   obj.type ="sedan"
   obj.color="blue"
   obj.make ="Bugatti"
   obj.model = "M"
   obj.license ="XX1234"
   obj.region ="CA"
   return obj

def generate_person_meta(data):
   obj = pyds.glist_get_nvds_person_object(data)
   obj.age = 45
   obj.cap = "none"
   obj.hair = "black"
   obj.gender = "male"
   obj.apparel= "formal"
   return obj

Generating event meta data

def generate_event_msg_meta(data, class_id):
    meta =pyds.glist_get_nvds_event_msg_meta(data)
    meta.sensorId = 0
    meta.placeId = 0
    meta.moduleId = 0
    meta.sensorStr = "sensor-0"
    meta.ts = pyds.alloc_buffer(MAX_TIME_STAMP_LEN + 1)
    pyds.generate_ts_rfc3339(meta.ts, MAX_TIME_STAMP_LEN)

    # This demonstrates how to attach custom objects.
    # Any custom object as per requirement can be generated and attached
    # like NvDsVehicleObject / NvDsPersonObject. Then that object should
    # be handled in payload generator library (nvmsgconv.cpp) accordingly.
    if(class_id==PGIE_CLASS_ID_VEHICLE):
        meta.type = pyds.NvDsEventType.NVDS_EVENT_MOVING
        meta.objType = pyds.NvDsObjectType.NVDS_OBJECT_TYPE_VEHICLE
        meta.objClassId = PGIE_CLASS_ID_VEHICLE
        obj = pyds.alloc_nvds_vehicle_object()
        obj = generate_vehicle_meta(obj)
        meta.extMsg = obj
        meta.extMsgSize = sys.getsizeof(pyds.NvDsVehicleObject);
    if(class_id == PGIE_CLASS_ID_PERSON):
        meta.type =pyds.NvDsEventType.NVDS_EVENT_ENTRY
        meta.objType = pyds.NvDsObjectType.NVDS_OBJECT_TYPE_PERSON;
        meta.objClassId = PGIE_CLASS_ID_PERSON
        obj = pyds.alloc_nvds_person_object()
        obj=generate_person_meta(obj)
        meta.extMsg = obj
        meta.extMsgSize = sys.getsizeof(pyds.NvDsPersonObject)
    return meta

探测指针的说明

Adding a probe to get meta data
osd_sink_pad_buffer_probe will extract metadata received on OSD sink pad
and update params for drawing rectangle, object information etc.

重要提示

a) probe() callbacks are synchronous and thus holds the buffer
(info.get_buffer()) from traversing the pipeline until user return.
b) loops inside probe() callback could be costly in python.
So users shall optimize according to their use-case.

代码

def osd_sink_pad_buffer_probe(pad,info,u_data):
    frame_number=0
    #Intiallizing object counter with 0.
    obj_counter = {
        PGIE_CLASS_ID_VEHICLE:0,
        PGIE_CLASS_ID_PERSON:0,
        PGIE_CLASS_ID_BICYCLE:0,
        PGIE_CLASS_ID_ROADSIGN:0
    }
    is_first_object=True
    gst_buffer = info.get_buffer()
    if not gst_buffer:
        print("Unable to get GstBuffer ")
        return

    # Retrieve batch metadata from the gst_buffer
    # Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
    # C address of gst_buffer as input, which is obtained with hash(gst_buffer)
    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
    if not batch_meta:
        return Gst.PadProbeReturn.OK
    l_frame = batch_meta.frame_meta_list
    while l_frame is not None:
        try:
            # Note that l_frame.data needs a cast to pyds.NvDsFrameMeta
            # The casting is done by pyds.glist_get_nvds_frame_meta()
            # The casting also keeps ownership of the underlying memory
            # in the C code, so the Python garbage collector will leave
            # it alone.
            frame_meta = pyds.glist_get_nvds_frame_meta(l_frame.data)
        except StopIteration:
            continue
        is_first_object = True;

        '''
        print("Frame Number is ", frame_meta.frame_num)
        print("Source id is ", frame_meta.source_id)
        print("Batch id is ", frame_meta.batch_id)
        print("Source Frame Width ", frame_meta.source_frame_width)
        print("Source Frame Height ", frame_meta.source_frame_height)
        print("Num object meta ", frame_meta.num_obj_meta)
        '''
        frame_number=frame_meta.frame_num
        l_obj=frame_meta.obj_meta_list
        while l_obj is not None:
            try:
                obj_meta=pyds.glist_get_nvds_object_meta(l_obj.data)
            except StopIteration:
                continue

            # Update the object text display
            txt_params=obj_meta.text_params
            if(txt_params.display_text):
                pyds.free_buffer(txt_params.display_text)

            txt_params.display_text = pgie_classes_str[obj_meta.class_id]

            obj_counter[obj_meta.class_id] += 1

            # Font , font-color and font-size
            txt_params.font_params.font_name = "Serif"
            txt_params.font_params.font_size = 10
            # set(red, green, blue, alpha); set to White
            txt_params.font_params.font_color.set(1.0, 1.0, 1.0, 1.0);

            # Text background color
            txt_params.set_bg_clr = 1
            # set(red, green, blue, alpha); set to Black
            txt_params.text_bg_clr.set(0.0, 0.0, 0.0, 1.0);

            # Ideally NVDS_EVENT_MSG_META should be attached to buffer by the
            # component implementing detection / recognition logic.
            # Here it demonstrates how to use / attach that meta data.
            if(is_first_object and not (frame_number%30)):
                # Frequency of messages to be send will be based on use case.
                # Here message is being sent for first object every 30 frames.

                # Allocating an NvDsEventMsgMeta instance and getting reference
                # to it. The underlying memory is not manged by Python so that
                # downstream plugins can access it. Otherwise the garbage collector
                # will free it when this probe exits.
                msg_meta=pyds.alloc_nvds_event_msg_meta()
                msg_meta.bbox.top =  obj_meta.rect_params.top
                msg_meta.bbox.left =  obj_meta.rect_params.left
                msg_meta.bbox.width = obj_meta.rect_params.width
                msg_meta.bbox.height = obj_meta.rect_params.height
                msg_meta.frameId = frame_number
                msg_meta.trackingId = long_to_int(obj_meta.object_id)
                msg_meta.confidence = obj_meta.confidence
                msg_meta = generate_event_msg_meta(msg_meta, obj_meta.class_id)
                user_event_meta = pyds.nvds_acquire_user_meta_from_pool(batch_meta)
                if(user_event_meta):
                    user_event_meta.user_meta_data = msg_meta;
                    user_event_meta.base_meta.meta_type = pyds.NvDsMetaType.NVDS_EVENT_MSG_META
                    # Setting callbacks in the event msg meta. The bindings layer
                    # will wrap these callables in C functions. Currently only one
                    # set of callbacks is supported.
                    pyds.set_user_copyfunc(user_event_meta, meta_copy_func)
                    pyds.set_user_releasefunc(user_event_meta, meta_free_func)
                    pyds.nvds_add_user_meta_to_frame(frame_meta, user_event_meta)
                else:
                    print("Error in attaching event meta to buffer\n")

                is_first_object = False
            try:
                l_obj=l_obj.next
            except StopIteration:
                break
        try:
            l_frame=l_frame.next
        except StopIteration:
            break

    print("Frame Number =",frame_number,"Vehicle Count =",obj_counter[PGIE_CLASS_ID_VEHICLE],"Person Count =",obj_counter[PGIE_CLASS_ID_PERSON])
    return Gst.PadProbeReturn.OK

Initializing GStreamer

GObject.threads_init()
Gst.init(None)

print("Creating Pipeline \n ")

pipeline = Gst.Pipeline()

if not pipeline:
   sys.stderr.write(" Unable to create Pipeline \n")

Creating different elements of the stream.Creating a source element for reading from a file

print("Creating Source \n ")
source = Gst.ElementFactory.make("filesrc", "file-source")
if not source:
    sys.stderr.write(" Unable to create Source \n")

Creating a h264 parser as the input file is an elementary h264 stream

print("Creating H264Parser \n")
h264parser = Gst.ElementFactory.make("h264parse", "h264-parser")
if not h264parser:
    sys.stderr.write(" Unable to create h264 parser \n")

Using nvdec_h264 for accelerated decoding on GPU

print("Creating Decoder \n")
decoder = Gst.ElementFactory.make("nvv4l2decoder", "nvv4l2-decoder")
if not decoder:
  sys.stderr.write(" Unable to create Nvv4l2 Decoder \n")

Creating a nvstreammux instance to form batches for one or more sources

streammux = Gst.ElementFactory.make("nvstreammux", "Stream-muxer")
if not streammux:
    sys.stderr.write(" Unable to create NvStreamMux \n")

Setting up nvinfer to run inference on the decoders output

Note: Behaviour of inference is set through the config file

pgie = Gst.ElementFactory.make("nvinfer", "primary-inference")
if not pgie:
    sys.stderr.write(" Unable to create pgie \n")

Using a converter to convert from NV12 to RGBA as required by nvosd

nvvidconv = Gst.ElementFactory.make("nvvideoconvert", "convertor")
if not nvvidconv:
   sys.stderr.write(" Unable to create nvvidconv \n")

Create OSD to draw on the converted RGBA buffer

nvosd = Gst.ElementFactory.make("nvdsosd", "onscreendisplay")
if not nvosd:
    sys.stderr.write(" Unable to create nvosd \n")

Create a message converter

msgconv=Gst.ElementFactory.make("nvmsgconv", "nvmsg-converter")
if not msgconv:
    sys.stderr.write(" Unable to create msgconv \n")

Create a message broker

msgbroker=Gst.ElementFactory.make("nvmsgbroker", "nvmsg-broker")
if not msgbroker:
    sys.stderr.write(" Unable to create msgbroker \n")

Create a tee

tee=Gst.ElementFactory.make("tee", "nvsink-tee")
if not tee:
    sys.stderr.write(" Unable to create tee \n")

Creating queues for tee

queue1=Gst.ElementFactory.make("queue", "nvtee-que1")
if not queue1:
    sys.stderr.write(" Unable to create queue1 \n")
    queue2=Gst.ElementFactory.make("queue", "nvtee-que2")
if not queue2:
    sys.stderr.write(" Unable to create queue2 \n")

Creating sink as required based on if display is required

if (no_display) :
   print("Creating FakeSink \n")
   sink = Gst.ElementFactory.make("fakesink", "fakesink")
   if not sink:
       sys.stderr.write(" Unable to create fakesink \n")
else:
   if is_aarch64():
       transform = Gst.ElementFactory.make("nvegltransform", "nvegl-transform")

   print("Creating EGLSink \n")
   sink = Gst.ElementFactory.make("nveglglessink", "nvvideo-renderer")
   if not sink:
       sys.stderr.write(" Unable to create egl sink \n")

Setting PGIE and Streammux properties

Suggested reading: dstest1_pgie_config.txt

代码

print("Playing file %s " %input_file)
source.set_property('location', input_file)
streammux.set_property('width', 1920)
streammux.set_property('height', 1080)
streammux.set_property('batch-size', 1)
streammux.set_property('batched-push-timeout', 4000000)
pgie.set_property('config-file-path', PGIE_CONFIG_FILE)
msgconv.set_property('config',MSCONV_CONFIG_FILE)
msgconv.set_property('payload-type', schema_type)
msgbroker.set_property('proto-lib', proto_lib)
msgbroker.set_property('conn-str', conn_str)
msgbroker.set_property('config', cfg_file)
if topic is not None:
    msgbroker.set_property('topic', topic)
msgbroker.set_property('sync', False)

Adding elements to pipeline

print("Adding elements to Pipeline \n")
pipeline.add(source)
pipeline.add(h264parser)
pipeline.add(decoder)
pipeline.add(streammux)
pipeline.add(pgie)
pipeline.add(nvvidconv)
pipeline.add(nvosd)
pipeline.add(tee)
pipeline.add(queue1)
pipeline.add(queue2)
pipeline.add(msgconv)
pipeline.add(msgbroker)
pipeline.add(sink)
if is_aarch64() and not no_display:
    pipeline.add(transform)

Linking elements of pipline

print("Linking elements in the Pipeline \n")
source.link(h264parser)
h264parser.link(decoder)

sinkpad = streammux.get_request_pad("sink_0")
if not sinkpad:
    sys.stderr.write(" Unable to get the sink pad of streammux \n")
srcpad = decoder.get_static_pad("src")
if not srcpad:
    sys.stderr.write(" Unable to get source pad of decoder \n")
srcpad.link(sinkpad)

streammux.link(pgie)
pgie.link(nvvidconv)
nvvidconv.link(nvosd)
nvosd.link(tee)
queue1.link(msgconv)
msgconv.link(msgbroker)
if is_aarch64() and not no_display:
    queue2.link(transform)
    transform.link(sink)
else:
    queue2.link(sink)
sink_pad=queue1.get_static_pad("sink")
tee_msg_pad=tee.get_request_pad('src_%u')
tee_render_pad=tee.get_request_pad("src_%u")
if not tee_msg_pad or not tee_render_pad:
    sys.stderr.write("Unable to get request pads\n")
tee_msg_pad.link(sink_pad)
sink_pad=queue2.get_static_pad("sink")
tee_render_pad.link(sink_pad)

Creating an event loop and feeding gstream bus messages to it

loop = GObject.MainLoop()
bus = pipeline.get_bus()
bus.add_signal_watch()
bus.connect ("message", bus_call, loop)

osdsinkpad = nvosd.get_static_pad("sink")
if not osdsinkpad:
    sys.stderr.write(" Unable to get sink pad of nvosd \n")

osdsinkpad.add_probe(Gst.PadProbeType.BUFFER, osd_sink_pad_buffer_probe, 0)

Starting the pipeline

print("Starting pipeline \n")

# start play back and listed to events
pipeline.set_state(Gst.State.PLAYING)
try:
    loop.run()
except:
    pass

Cleaning up post successful run

pyds.unset_callback_funcs()
pipeline.set_state(Gst.State.NULL)