原文:deepstream_python_apps/notebooks/deepstream_test_4.ipynb
代码比较难看,但是还好作者留下了NOTEBOOK
Deepstream Test4 description
TEST4建立在TEST1并添加展示了功能:
1、在pipeline中使用了 “nvmsgconv” 和 “nvmsgbroker” 插件
2、创建 NVDS_META_EVENT_MSG 元数据并将其加入缓存(buffer).
3、对不同物体使用NVDS_META_EVENT_MSG
4、如果元数据通过“extMsg”字段扩展,提供了复制/释放功能。
对两个插件的说明太长了,直接机翻
“nvmsgconv”插件从缓冲区使用NVDS\u META\u EVENT\u MSG类型的元数据,并以Json格式生成“DeepStream Schema”负载。模式的静态属性以键值对的形式从配置文件中读取。检查dstest4\msgconv_配置.txt供参考。生成的负载作为NVDS\u META\u负载类型元数据附加到缓冲区。
“nvmsgbroker”插件从缓冲区提取NVDS\u META\u负载类型的元数据,并使用协议适配器api将该负载发送到服务器。
为不同类型的对象生成自定义元数据:除了NvDsEventMsgMeta结构中提供的公共字段外,用户还可以创建自定义对象并作为NVDS\u META\u EVENT\u MSG元数据附加到缓冲区。为此,NvDsEventMsgMeta提供了“extMsg”和“extMsgSize”字段。用户可以创建自定义结构,填充该结构,并将该结构的指针指定为“extMsg”,并相应地设置“extMsgSize”。如果自定义对象包含不能简单复制的字段,那么用户还应该提供复制和释放这些对象的功能。
附原文
“nvmsgconv” plugin uses NVDS_META_EVENT_MSG type of metadata from the buffer and generates the “DeepStream Schema” payload in Json format. Static properties of schema are read from configuration file in the form of key-value pair. Check dstest4_msgconv_config.txt for reference. Generated payload is attached as NVDS_META_PAYLOAD type metadata to the buffer.
“nvmsgbroker” plugin extracts NVDS_META_PAYLOAD type of metadata from the buffer and sends that payload to the server using protocol adaptor APIs.
Generating custom metadata for different type of objects: In addition to common fields provided in NvDsEventMsgMeta structure, user can also create custom objects and attach to buffer as NVDS_META_EVENT_MSG metadata. To do that NvDsEventMsgMeta provides “extMsg” and “extMsgSize” fields. User can create custom structure, fill that structure and assign the pointer of that structure as “extMsg” and set the “extMsgSize” accordingly. If custom object contains fields that can’t be simply mem copied then user should also provide function to copy and free those objects.
Installing necessary libraries (kafka)
In [ ]:
!sudo apt-get install libglib2.0 libglib2.0-dev
!sudo apt-get install libjansson4 libjansson-dev
!sudo apt-get install librdkafka1=0.11.3-1build1
Preparing the necessary arguments
cfg_file = Path to adaptor config file
input_file = Path to input x264 stream
proto_lib = Absolute path to adaptor library
conn_str = Connection string of backend server. Optional if it is part of config file.
topic = Name of message topic. Optional if it is part of connection string or config file.
no_display = To disable display. Default False schema_type = Type of minimal schema. 0= Full. 1= Minimal. Default =0
cfg_file = "cfg_kafka.txt" input_file = "" proto_lib = "/home/nvidia/Downloads/deepstream/sources/libs/kafka_protocol_adaptor/" conn_str = "host;port;topic" topic = "" no_display = False schema_type = 0
模块说明
Importing necessary libraries
import sys
sys.path.append('../')
sys.path.append('/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/')
import gi
gi.require_version('Gst', '1.0')
from gi.repository import GObject, Gst
from gi.repository import GLib
import sys
import platform
from optparse import OptionParser
from common.is_aarch_64 import is_aarch64
from common.bus_call import bus_call
from common.utils import long_to_int
import pyds
Declaring class label ids and other meta data requirements
MAX_DISPLAY_LEN=64
MAX_TIME_STAMP_LEN=32
PGIE_CLASS_ID_VEHICLE = 0
PGIE_CLASS_ID_BICYCLE = 1
PGIE_CLASS_ID_PERSON = 2
PGIE_CLASS_ID_ROADSIGN = 3
MUXER_OUTPUT_WIDTH=1920
MUXER_OUTPUT_HEIGHT=1080
MUXER_BATCH_TIMEOUT_USEC=4000000
input_file = "/home/nvidia/Downloads/deepstream/sources/python/apps/sample_720p.h264"
schema_type = 0
frame_number = 0
proto_lib = "/home/nvidia/Downloads/deepstream/sources/libs/kafka_protocol_adaptor/"
conn_str="localhost;2181;testTopic"
cfg_file = "cfg_kafka.txt"
topic = None
no_display = False
Necessary config files and meta data class labels
PGIE_CONFIG_FILE="dstest4_pgie_config.txt"
MSCONV_CONFIG_FILE="dstest4_msgconv_config.txt"
pgie_classes_str=["Vehicle", "TwoWheeler", "Person","Roadsign"]
Callback function for deep-copying an NvDsEventMsgMeta struct
def meta_copy_func(data,user_data):
# Cast data to pyds.NvDsUserMeta
user_meta=pyds.glist_get_nvds_user_meta(data)
src_meta_data=user_meta.user_meta_data
# Cast src_meta_data to pyds.NvDsEventMsgMeta
srcmeta=pyds.glist_get_nvds_event_msg_meta(src_meta_data)
# Duplicate the memory contents of srcmeta to dstmeta
# First use pyds.get_ptr() to get the C address of srcmeta, then
# use pyds.memdup() to allocate dstmeta and copy srcmeta into it.
# pyds.memdup returns C address of the allocated duplicate.
dstmeta_ptr=pyds.memdup(pyds.get_ptr(srcmeta), sys.getsizeof(pyds.NvDsEventMsgMeta))
# Cast the duplicated memory to pyds.NvDsEventMsgMeta
dstmeta=pyds.glist_get_nvds_event_msg_meta(dstmeta_ptr)
# Duplicate contents of ts field. Note that reading srcmeat.ts
# returns its C address. This allows to memory operations to be
# performed on it.
dstmeta.ts=pyds.memdup(srcmeta.ts, MAX_TIME_STAMP_LEN+1)
# Copy the sensorStr. This field is a string property.
# The getter (read) returns its C address. The setter (write)
# takes string as input, allocates a string buffer and copies
# the input string into it.
# pyds.get_string() takes C address of a string and returns
# the reference to a string object and the assignment inside the binder copies content.
dstmeta.sensorStr=pyds.get_string(srcmeta.sensorStr)
if(srcmeta.objSignature.size>0):
dstmeta.objSignature.signature=pyds.memdup(srcmeta.objSignature.signature,srcMeta.objSignature.size)
dstmeta.objSignature.size = srcmeta.objSignature.size;
if(srcmeta.extMsgSize>0):
if(srcmeta.objType==pyds.NvDsObjectType.NVDS_OBJECT_TYPE_VEHICLE):
srcobj = pyds.glist_get_nvds_vehicle_object(srcmeta.extMsg);
obj = pyds.alloc_nvds_vehicle_object();
obj.type=pyds.get_string(srcobj.type)
obj.make=pyds.get_string(srcobj.make)
obj.model=pyds.get_string(srcobj.model)
obj.color=pyds.get_string(srcobj.color)
obj.license = pyds.get_string(srcobj.license)
obj.region = pyds.get_string(srcobj.region)
dstmeta.extMsg = obj;
dstmeta.extMsgSize = sys.getsizeof(pyds.NvDsVehicleObject)
if(srcmeta.objType==pyds.NvDsObjectType.NVDS_OBJECT_TYPE_PERSON):
srcobj = pyds.glist_get_nvds_person_object(srcmeta.extMsg);
obj = pyds.alloc_nvds_person_object()
obj.age = srcobj.age
obj.gender = pyds.get_string(srcobj.gender);
obj.cap = pyds.get_string(srcobj.cap)
obj.hair = pyds.get_string(srcobj.hair)
obj.apparel = pyds.get_string(srcobj.apparel);
dstmeta.extMsg = obj;
dstmeta.extMsgSize = sys.getsizeof(pyds.NvDsVehicleObject);
return dstmeta
Callback function for freeing an NvDsEventMsgMeta instance
def meta_free_func(data,user_data):
user_meta=pyds.glist_get_nvds_user_meta(data)
srcmeta=pyds.glist_get_nvds_event_msg_meta(user_meta.user_meta_data)
# pyds.free_buffer takes C address of a buffer and frees the memory
# It's a NOP if the address is NULL
pyds.free_buffer(srcmeta.ts)
pyds.free_buffer(srcmeta.sensorStr)
if(srcmeta.objSignature.size > 0):
pyds.free_buffer(srcmeta.objSignature.signature);
srcmeta.objSignature.size = 0
if(srcmeta.extMsgSize > 0):
if(srcmeta.objType == pyds.NvDsObjectType.NVDS_OBJECT_TYPE_VEHICLE):
obj =pyds.glist_get_nvds_vehicle_object(srcmeta.extMsg)
pyds.free_buffer(obj.type);
pyds.free_buffer(obj.color);
pyds.free_buffer(obj.make);
pyds.free_buffer(obj.model);
pyds.free_buffer(obj.license);
pyds.free_buffer(obj.region);
if(srcmeta.objType == pyds.NvDsObjectType.NVDS_OBJECT_TYPE_PERSON):
obj = pyds.glist_get_nvds_person_object(srcmeta.extMsg);
pyds.free_buffer(obj.gender);
pyds.free_buffer(obj.cap);
pyds.free_buffer(obj.hair);
pyds.free_buffer(obj.apparel);
pyds.free_gbuffer(srcmeta.extMsg);
srcmeta.extMsgSize = 0;
Generating meta data
def generate_vehicle_meta(data):
obj = pyds.glist_get_nvds_vehicle_object(data);
obj.type ="sedan"
obj.color="blue"
obj.make ="Bugatti"
obj.model = "M"
obj.license ="XX1234"
obj.region ="CA"
return obj
def generate_person_meta(data):
obj = pyds.glist_get_nvds_person_object(data)
obj.age = 45
obj.cap = "none"
obj.hair = "black"
obj.gender = "male"
obj.apparel= "formal"
return obj
Generating event meta data
def generate_event_msg_meta(data, class_id):
meta =pyds.glist_get_nvds_event_msg_meta(data)
meta.sensorId = 0
meta.placeId = 0
meta.moduleId = 0
meta.sensorStr = "sensor-0"
meta.ts = pyds.alloc_buffer(MAX_TIME_STAMP_LEN + 1)
pyds.generate_ts_rfc3339(meta.ts, MAX_TIME_STAMP_LEN)
# This demonstrates how to attach custom objects.
# Any custom object as per requirement can be generated and attached
# like NvDsVehicleObject / NvDsPersonObject. Then that object should
# be handled in payload generator library (nvmsgconv.cpp) accordingly.
if(class_id==PGIE_CLASS_ID_VEHICLE):
meta.type = pyds.NvDsEventType.NVDS_EVENT_MOVING
meta.objType = pyds.NvDsObjectType.NVDS_OBJECT_TYPE_VEHICLE
meta.objClassId = PGIE_CLASS_ID_VEHICLE
obj = pyds.alloc_nvds_vehicle_object()
obj = generate_vehicle_meta(obj)
meta.extMsg = obj
meta.extMsgSize = sys.getsizeof(pyds.NvDsVehicleObject);
if(class_id == PGIE_CLASS_ID_PERSON):
meta.type =pyds.NvDsEventType.NVDS_EVENT_ENTRY
meta.objType = pyds.NvDsObjectType.NVDS_OBJECT_TYPE_PERSON;
meta.objClassId = PGIE_CLASS_ID_PERSON
obj = pyds.alloc_nvds_person_object()
obj=generate_person_meta(obj)
meta.extMsg = obj
meta.extMsgSize = sys.getsizeof(pyds.NvDsPersonObject)
return meta
探测指针的说明
Adding a probe to get meta data
osd_sink_pad_buffer_probe will extract metadata received on OSD sink pad
and update params for drawing rectangle, object information etc.
重要提示
a) probe() callbacks are synchronous and thus holds the buffer
(info.get_buffer()) from traversing the pipeline until user return.
b) loops inside probe() callback could be costly in python.
So users shall optimize according to their use-case.
代码
def osd_sink_pad_buffer_probe(pad,info,u_data):
frame_number=0
#Intiallizing object counter with 0.
obj_counter = {
PGIE_CLASS_ID_VEHICLE:0,
PGIE_CLASS_ID_PERSON:0,
PGIE_CLASS_ID_BICYCLE:0,
PGIE_CLASS_ID_ROADSIGN:0
}
is_first_object=True
gst_buffer = info.get_buffer()
if not gst_buffer:
print("Unable to get GstBuffer ")
return
# Retrieve batch metadata from the gst_buffer
# Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
# C address of gst_buffer as input, which is obtained with hash(gst_buffer)
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
if not batch_meta:
return Gst.PadProbeReturn.OK
l_frame = batch_meta.frame_meta_list
while l_frame is not None:
try:
# Note that l_frame.data needs a cast to pyds.NvDsFrameMeta
# The casting is done by pyds.glist_get_nvds_frame_meta()
# The casting also keeps ownership of the underlying memory
# in the C code, so the Python garbage collector will leave
# it alone.
frame_meta = pyds.glist_get_nvds_frame_meta(l_frame.data)
except StopIteration:
continue
is_first_object = True;
'''
print("Frame Number is ", frame_meta.frame_num)
print("Source id is ", frame_meta.source_id)
print("Batch id is ", frame_meta.batch_id)
print("Source Frame Width ", frame_meta.source_frame_width)
print("Source Frame Height ", frame_meta.source_frame_height)
print("Num object meta ", frame_meta.num_obj_meta)
'''
frame_number=frame_meta.frame_num
l_obj=frame_meta.obj_meta_list
while l_obj is not None:
try:
obj_meta=pyds.glist_get_nvds_object_meta(l_obj.data)
except StopIteration:
continue
# Update the object text display
txt_params=obj_meta.text_params
if(txt_params.display_text):
pyds.free_buffer(txt_params.display_text)
txt_params.display_text = pgie_classes_str[obj_meta.class_id]
obj_counter[obj_meta.class_id] += 1
# Font , font-color and font-size
txt_params.font_params.font_name = "Serif"
txt_params.font_params.font_size = 10
# set(red, green, blue, alpha); set to White
txt_params.font_params.font_color.set(1.0, 1.0, 1.0, 1.0);
# Text background color
txt_params.set_bg_clr = 1
# set(red, green, blue, alpha); set to Black
txt_params.text_bg_clr.set(0.0, 0.0, 0.0, 1.0);
# Ideally NVDS_EVENT_MSG_META should be attached to buffer by the
# component implementing detection / recognition logic.
# Here it demonstrates how to use / attach that meta data.
if(is_first_object and not (frame_number%30)):
# Frequency of messages to be send will be based on use case.
# Here message is being sent for first object every 30 frames.
# Allocating an NvDsEventMsgMeta instance and getting reference
# to it. The underlying memory is not manged by Python so that
# downstream plugins can access it. Otherwise the garbage collector
# will free it when this probe exits.
msg_meta=pyds.alloc_nvds_event_msg_meta()
msg_meta.bbox.top = obj_meta.rect_params.top
msg_meta.bbox.left = obj_meta.rect_params.left
msg_meta.bbox.width = obj_meta.rect_params.width
msg_meta.bbox.height = obj_meta.rect_params.height
msg_meta.frameId = frame_number
msg_meta.trackingId = long_to_int(obj_meta.object_id)
msg_meta.confidence = obj_meta.confidence
msg_meta = generate_event_msg_meta(msg_meta, obj_meta.class_id)
user_event_meta = pyds.nvds_acquire_user_meta_from_pool(batch_meta)
if(user_event_meta):
user_event_meta.user_meta_data = msg_meta;
user_event_meta.base_meta.meta_type = pyds.NvDsMetaType.NVDS_EVENT_MSG_META
# Setting callbacks in the event msg meta. The bindings layer
# will wrap these callables in C functions. Currently only one
# set of callbacks is supported.
pyds.set_user_copyfunc(user_event_meta, meta_copy_func)
pyds.set_user_releasefunc(user_event_meta, meta_free_func)
pyds.nvds_add_user_meta_to_frame(frame_meta, user_event_meta)
else:
print("Error in attaching event meta to buffer\n")
is_first_object = False
try:
l_obj=l_obj.next
except StopIteration:
break
try:
l_frame=l_frame.next
except StopIteration:
break
print("Frame Number =",frame_number,"Vehicle Count =",obj_counter[PGIE_CLASS_ID_VEHICLE],"Person Count =",obj_counter[PGIE_CLASS_ID_PERSON])
return Gst.PadProbeReturn.OK
Initializing GStreamer
GObject.threads_init()
Gst.init(None)
print("Creating Pipeline \n ")
pipeline = Gst.Pipeline()
if not pipeline:
sys.stderr.write(" Unable to create Pipeline \n")
Creating different elements of the stream.Creating a source element for reading from a file
print("Creating Source \n ")
source = Gst.ElementFactory.make("filesrc", "file-source")
if not source:
sys.stderr.write(" Unable to create Source \n")
Creating a h264 parser as the input file is an elementary h264 stream
print("Creating H264Parser \n")
h264parser = Gst.ElementFactory.make("h264parse", "h264-parser")
if not h264parser:
sys.stderr.write(" Unable to create h264 parser \n")
Using nvdec_h264 for accelerated decoding on GPU
print("Creating Decoder \n")
decoder = Gst.ElementFactory.make("nvv4l2decoder", "nvv4l2-decoder")
if not decoder:
sys.stderr.write(" Unable to create Nvv4l2 Decoder \n")
Creating a nvstreammux instance to form batches for one or more sources
streammux = Gst.ElementFactory.make("nvstreammux", "Stream-muxer")
if not streammux:
sys.stderr.write(" Unable to create NvStreamMux \n")
Setting up nvinfer to run inference on the decoders output
Note: Behaviour of inference is set through the config file
pgie = Gst.ElementFactory.make("nvinfer", "primary-inference")
if not pgie:
sys.stderr.write(" Unable to create pgie \n")
Using a converter to convert from NV12 to RGBA as required by nvosd
nvvidconv = Gst.ElementFactory.make("nvvideoconvert", "convertor")
if not nvvidconv:
sys.stderr.write(" Unable to create nvvidconv \n")
Create OSD to draw on the converted RGBA buffer
nvosd = Gst.ElementFactory.make("nvdsosd", "onscreendisplay")
if not nvosd:
sys.stderr.write(" Unable to create nvosd \n")
Create a message converter
msgconv=Gst.ElementFactory.make("nvmsgconv", "nvmsg-converter")
if not msgconv:
sys.stderr.write(" Unable to create msgconv \n")
Create a message broker
msgbroker=Gst.ElementFactory.make("nvmsgbroker", "nvmsg-broker")
if not msgbroker:
sys.stderr.write(" Unable to create msgbroker \n")
Create a tee
tee=Gst.ElementFactory.make("tee", "nvsink-tee")
if not tee:
sys.stderr.write(" Unable to create tee \n")
Creating queues for tee
queue1=Gst.ElementFactory.make("queue", "nvtee-que1")
if not queue1:
sys.stderr.write(" Unable to create queue1 \n")
queue2=Gst.ElementFactory.make("queue", "nvtee-que2")
if not queue2:
sys.stderr.write(" Unable to create queue2 \n")
Creating sink as required based on if display is required
if (no_display) :
print("Creating FakeSink \n")
sink = Gst.ElementFactory.make("fakesink", "fakesink")
if not sink:
sys.stderr.write(" Unable to create fakesink \n")
else:
if is_aarch64():
transform = Gst.ElementFactory.make("nvegltransform", "nvegl-transform")
print("Creating EGLSink \n")
sink = Gst.ElementFactory.make("nveglglessink", "nvvideo-renderer")
if not sink:
sys.stderr.write(" Unable to create egl sink \n")
Setting PGIE and Streammux properties
Suggested reading: dstest1_pgie_config.txt
代码
print("Playing file %s " %input_file)
source.set_property('location', input_file)
streammux.set_property('width', 1920)
streammux.set_property('height', 1080)
streammux.set_property('batch-size', 1)
streammux.set_property('batched-push-timeout', 4000000)
pgie.set_property('config-file-path', PGIE_CONFIG_FILE)
msgconv.set_property('config',MSCONV_CONFIG_FILE)
msgconv.set_property('payload-type', schema_type)
msgbroker.set_property('proto-lib', proto_lib)
msgbroker.set_property('conn-str', conn_str)
msgbroker.set_property('config', cfg_file)
if topic is not None:
msgbroker.set_property('topic', topic)
msgbroker.set_property('sync', False)
Adding elements to pipeline
print("Adding elements to Pipeline \n")
pipeline.add(source)
pipeline.add(h264parser)
pipeline.add(decoder)
pipeline.add(streammux)
pipeline.add(pgie)
pipeline.add(nvvidconv)
pipeline.add(nvosd)
pipeline.add(tee)
pipeline.add(queue1)
pipeline.add(queue2)
pipeline.add(msgconv)
pipeline.add(msgbroker)
pipeline.add(sink)
if is_aarch64() and not no_display:
pipeline.add(transform)
Linking elements of pipline
print("Linking elements in the Pipeline \n")
source.link(h264parser)
h264parser.link(decoder)
sinkpad = streammux.get_request_pad("sink_0")
if not sinkpad:
sys.stderr.write(" Unable to get the sink pad of streammux \n")
srcpad = decoder.get_static_pad("src")
if not srcpad:
sys.stderr.write(" Unable to get source pad of decoder \n")
srcpad.link(sinkpad)
streammux.link(pgie)
pgie.link(nvvidconv)
nvvidconv.link(nvosd)
nvosd.link(tee)
queue1.link(msgconv)
msgconv.link(msgbroker)
if is_aarch64() and not no_display:
queue2.link(transform)
transform.link(sink)
else:
queue2.link(sink)
sink_pad=queue1.get_static_pad("sink")
tee_msg_pad=tee.get_request_pad('src_%u')
tee_render_pad=tee.get_request_pad("src_%u")
if not tee_msg_pad or not tee_render_pad:
sys.stderr.write("Unable to get request pads\n")
tee_msg_pad.link(sink_pad)
sink_pad=queue2.get_static_pad("sink")
tee_render_pad.link(sink_pad)
Creating an event loop and feeding gstream bus messages to it
loop = GObject.MainLoop()
bus = pipeline.get_bus()
bus.add_signal_watch()
bus.connect ("message", bus_call, loop)
osdsinkpad = nvosd.get_static_pad("sink")
if not osdsinkpad:
sys.stderr.write(" Unable to get sink pad of nvosd \n")
osdsinkpad.add_probe(Gst.PadProbeType.BUFFER, osd_sink_pad_buffer_probe, 0)
Starting the pipeline
print("Starting pipeline \n")
# start play back and listed to events
pipeline.set_state(Gst.State.PLAYING)
try:
loop.run()
except:
pass
Cleaning up post successful run
pyds.unset_callback_funcs()
pipeline.set_state(Gst.State.NULL)