基于Qualcomm Robotics RB5边缘侧AI应用—人脸/口罩/人员计数目标检测

csdnsqst0050

已于 2024-01-11 14:03:19 修改

阅读量1.1k

点赞数 26

分类专栏： SNPE AI 文章标签： Qualcomm AI SNPE

于 2024-01-11 10:39:05 首次发布

本文链接：https://blog.csdn.net/csdnsqst0050/article/details/135520850

版权

AI 同时被 2 个专栏收录

24 篇文章 4 订阅

订阅专栏

SNPE

4 篇文章 0 订阅

订阅专栏

所需使用硬件设备

An Ubuntu 20.04 PC
Qualcomm Robotics RB5 Development kit: https://developer.qualcomm.com/qualcomm-robotics-rb5-kithttps://developer.qualcomm.com/qualcomm-robotics-rb5-kit
A USB camera
A display monitor

我们将讨论使用YoloV5可以实现的各种对象检测解决方案：

- 人员入侵检测

- 头盔检测

- 火灾探测

- 施工现场裂缝检测

- 垃圾检测

- 人数统计

- 盒子损坏检测

为了减少和减轻严重事故，我们在仓库和建筑工地等各个行业开发了一套安全解决方案。这些解决方案将从IP摄像头获取输入，并在高通®Robotics RB5开发的Edge AI设备上执行实时输入流分析。这些解决方案提供了一个端到端的应用程序，可以快速集成到新的构建系统中。

1. 前提条件

One x86 Host Workstation. Operating System Ubuntu 20.04， Python 3.6.9
Qualcomm Robotics RB5 Development kit: https://developer.qualcomm.com/qualcomm-robotics-rb5-kit

2. x86 Host Installation and Deep Learning Model Compile

应用程序中使用的模型需要使用数据集进行训练。然后将模型转换为dlc以在应用程序中运行。

2.1 Qualcomm® Neural Processing SDK Installation

Download Software Development Kit (SDK) from Qualcomm Neural Processing SDK for AI Tools & Resources Archive - Qualcomm Developer Network

高通神经处理SDK提供了用于模型转换（onnx到dlc）、模型量化和执行的工具。请参阅SDK中详细文档中给出的步骤进行安装. Snapdragon Neural Processing Engine SDK: Main Page

2.2 准备Yolov5 AI模型

按照 ModelPreperation.md 导出和编译Yolov5模型

下载人员检测预训练YoloV5模型https://github.com/ultralytics/yolov5/releases/download/v6.0/yolov5s.pt

从Github下载Yolov5火灾和烟雾AI模型参考https://github.com/gengyanlei/fire-smoke-detect-yolov4/blob/master/yolov5/best.pt

对于头盔和头部检测，需要在数据集上训练模型。头盔和头部检测数据集可以从Kaggle下载。Hard Head Detection with YOLOv5 | Kaggle

**2.3 Export *.pt file to ONNX file**

建议安装Python 3.6.9版本。下载Yolov5 v6.0源代码

wget https://github.com/ultralytics/yolov5/archive/refs/tags/v6.0.tar.gz

tar -zxvf v6.0.tar.gz

cd yolov5-6.0

wget https://github.com/ultralytics/yolov5/releases/download/v6.0/yolov5s.pt

wget https://github.com/gengyanlei/fire-smoke-detect-yolov4/blob/master/yolov5/best.pt

Python要求：

python 3.6.9
protobuf==3.17.3
onnx==1.9.0
onnx-simplifier==0.2.6
onnxoptimizer==0.2.6
onnxruntime==1.1.0
numpy==1.16.5
torch==1.10.0
torchvision==0.11.1

sudo apt install python3.6 python3.6-venv build-essential make python3-dev python3.6-dev protobuf-compiler libprotobuf-dev

python3.6 -m pip install --upgrade pip

python3.6 -m pip install -r requirements.txt

python3.6 -m pip install coremltools>=4.1 onnx==1.9.0 scikit-learn==0.19.2 onnxruntime==1.1.0 onnx-simplifier==0.2.6 onnxoptimizer==0.2.6

将YoloV5导出到ONNX：

python3.6 export.py --weights yolov5m.pt --optimize --opset 11 --simplify --include onnx -imgsz [416,416]

3. 将Onnx模型转换为DLC

3.1 设置高通神经处理SDK

需要设置SNPE将onnx模型转换为量化的dlc，请使用提供的链接按照说明设置神经处理SDK。请在整个演示过程中使用相同版本的SNPE https://developer.qualcomm.com/docs/snpe/setup.html

export SNPE_ROOT=<path to snpe folder>/snpe-1.68.0.3932

export ANDROID_NDK_ROOT=<path to android ndk folder>

从pip查找onnx安装路径

python3.6 -m pip show onnx

在上述命令的输出中查找“Location:”行

export ONNX_DIR=<path from Location line>/onnx

为snpe设置onnx环境

cd $SNPE_ROOT

source bin/envsetup.sh -o $ONNX_DIR

SNPE目前不支持5D运算符。它要求在转换命令中的5D重塑之前指定输出节点。可以在Netron中检查输出节点。要检查输出层节点，请在Netron应用程序中打开模型，然后单击Conv层。

在附加的快照中，5D之前的输出节点为onnx:::443 (Conv_271), 496 (Conv_305) and 549 (Conv_339)

此实现具有以下功能:

anchorBoxProcess: Get raw data from out nodes before 5D (Conv_271, Conv_305, Conv_339), convert to meaning data (scores, class, bounding boxes).
doNMS: (non-max suppression): remove overlap boxes
ShowDetectionOverlay: Overlay detection result at output video/Image

3.2 转换为DLC

snpe-onnx-to-dlc -i yolov5m.onnx --out_node <output name 1> --out_node <output name 2> --out_node <output name 3>

与上面的屏幕截图相对应的示例：

snpe-onnx-to-dlc -i yolov5m.onnx --out_node 443 --out_node 496 --out_node 549

3.3 生成AIP/DSP的量化模型

使用设备中安装的相同高通®神经处理SDK版本。请参考模型目录中给出的inputlist.txt文件。根据您的yolov5s.onnx模型创建您自己的inputlist.txt文件。我们需要更新inputlist.txt中的所有输出名称

snpe-dlc-quantize --input_dlc=yolov5m.dlc --input_list=inputlist.txt --output_dlc=yolov5m_quant.dlc --enable_hta --use_enhanced_quantizer --optimizations cle --axis_quant

模型量化需要input.raw文件。使用以下python命令创建一个示例input.raw文件。

import numpy as np

((np.random.random((1,3,640,640)).astype(np.float32))).tofile("input.raw")

Please read Snapdragon Neural Processing Engine SDK: Quantized vs Non-Quantized Models to know more about model quantization.

4. X86 Host RTSP Stream Installation

在本节中，演示如何准备测试视频并设置live555 RTSP流媒体服务器以模拟IP摄像机。安装步骤在X86主机中运行。

4.1 准备测试视频

如果测试视频是mp4、mkv或其他格式，请准备一个测试视频。需要转换为H264原始视频。以下步骤演示如何将MP4转换为H264原始视频

wget https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/1080/Big_Buck_Bunny_1080_10s_1MB.mp4

sudo apt install ffmpeg

ffmpeg -i Big_Buck_Bunny_1080_10s_1MB.mp4 -f h264 -vcodec libx264 Big_Buck_Bunny_1080_10s_1MB.264

4.2 Live555服务器安装

wget http://www.live555.com/liveMedia/public/live555-latest.tar.gz

tar -zxvf live555-latest.tar.gz

cd live/

./genMakefiles linux-64bit

make -j4

cd ..

将测试视频复制到mediaServer文件夹

cp Big_Buck_Bunny_1080_10s_1MB.264 ./live/mediaServer

cd ./live/mediaServer

./live555MediaServer

rtsp://192.168.4.111:8554/ is the rtsp url.

"192.168.4.111" is RTSP server IP address
8554 is the default port
is the video file name under mediaServer folder In this case, url "rtsp://192.168.4.111:8554/Big_Buck_Bunny_1080_10s_1MB.264" is the video address.

4.3 验证live555服务器

在Windows桌面上下载并安装VLC媒体播放器VLC: Official site - Free multimedia solutions for all OS! - VideoLAN

Launch VLC player, choose "Media->Open Network Stream" input RTSP url rtsp://192.168.4.111:8554/Big_Buck_Bunny_1080_10s_1MB.264

单击播放以测试live555服务器是否正常工作。注意：确保网络地址可访问

配置、构建和运行此应用程序

1. 克隆存储库

adb shell

cd /home/

apt install git

git clone https://github.com/quic/sample-apps-for-robotics-platforms.git

cd sample-apps-for-robotics-platforms/RB5/linux_kernel_5_x/AI-ML-apps/AI-Vision-Solutions/

2. 更新应用程序配置

所有解决方案的配置在data/config.json文件中进行了描述。应更新此配置文件以选择所需的解决方案、模型配置和输入/输出流。应用程序可以将rtsp/camera流作为输入，并将输出转储到mp4或hdmi监视器。

Table 1-1 show all the configuration items:

输入配置

key	Value	Description
input-config-name	string	Name of the input config
stream-type	string	Input stream type camera or rtsp
stream-width	int	Width of the input stream
stream-height	int	Height of the input stream
SkipFrame	int	Number of frames to be skipped, default value:1
camera-url	string	rtsp stream path if the input stream is rtsp

模型配置

key	Value	Description
model-name	string	Name of the model
model-type	string	Type of the mode, it is yolov5
model-path	string	Path of the dlc file
label-path	string	Path of the label file
runtime	string	SNPE Runtime (GPU, CPU, DSP)
nms-threshold	float	NMS threshold
conf-threshold	float	Confidence threshold
labels	int	Number of labels + 5
input-layers	string	Name of the input layers
output-layers	string	Name of the output layers
output-tensors	string	Name of the output tensors

解决方案配置

key	Value	Description
solution-name	string	Name of the Solution
model-name	string	Name of the model configuration to be used
input-config-name	string	Name of the Input configuration to be used
Enable	bool	1 to Enable and 0 to Disable the solution
output-type	string	Filesink to save the output in mp4
		Wayland if display the output on hdmi monitor
output-path	string	Path of the output, Enabled if output type
		is filesink

Example 1: 从相机获取输入流并在hdmi监视器上输出的配置

    "input-configs":[

            "input-config-name":"camera",

            "stream-type":"camera",

            "stream-width":1280,

            "stream-height":720,

            "SkipFrame":1,

            "fps-n":30,

            "fps-d":1

},

],

     "model-configs":[

            "model-name":"yolov5s-1",

            "model-type":"yolov5",

            "model-path":"../models/people_intrusion_detection.dlc",

            "label-path":"../data/coco_label.txt",

            "runtime":"DSP",

            "labels":85,

            "grids":25200,

            "nms-threshold":0.5,

            "conf-threshold":0.5,

            "input-layers":[

                "images"

],

            "output-layers":[

                "Conv_271",

                "Conv_305",

                "Conv_339"

],

            "output-tensors":[

                "443",

                "496",

                "549"

],

            "global-threshold":0.2

},

],

     "solution-configs":[

            "solution-name":"people-intrusion-detection",

            "model-name":"yolov5s-1",

            "input-config-name":"camera",

            "Enable":0,

            "strict_area_x":10,

            "strict_area_y":10,

            "strict_area_w":1000,

            "strict_area_h":1000,

            "output-type":"wayland",

},

],

Example 2: 设备上rtsp输入流和输出的配置

"input-configs":[

        "input-config-name":"rtsp3",

        "stream-type":"rtsp",

        "camera-url":"rtsp://10.147.243.253:8554/crack_video.264",

        "SkipFrame":1

},

"model-configs":[

        "model-name":"yolov5s-3",

        "model-type":"yolov5",

        "model-path":"../models/crack_detection.dlc",

        "label-path":"../data/crack_labels.txt",

        "runtime":"DSP",

        "nms-threshold":0.1,

        "conf-threshold":0.1,

        "labels":6,

        "grids":25200,

        "input-layers":[

            "images"

],

        "output-layers":[

            "/model.24/m.0/Conv",

            "/model.24/m.1/Conv",

            "/model.24/m.2/Conv"

],

        "output-tensors":[

            "/model.24/m.0/Conv_output_0",

            "/model.24/m.1/Conv_output_0",

            "/model.24/m.2/Conv_output_0"

],

        "global-threshold":0.2

},

"solution-configs":[

        "solution-name":"crack-solution",

        "model-name":"yolov5s-3",

        "input-config-name":"rtsp3",

        "Enable":1,

        "output-type":"filesink",

        "output-path":"/root/crack.mp4"

},

Use model-name and input-config-name to select model and input stream respectively.

3. 模型集成

将model推送到应用程序中的model目录中，并更新config.json文件。

更新输出层和输出张量。要检查输出层和输出张量节点，请在Netron应用程序中打开模型，然后单击图像中提到的Conv层。

在yolov5s.onnx中，5D之前的输出层是onnx::443 (Conv_271), 496 (Conv_305) and 549 (Conv_339) and 输出张量是 Conv_271, Conv_305 and Conv_339.

  "model-configs":[

          "model-name":"yolov5s-1", --> Add model name here. It should match with the model name in solution config

          "model-type":"yolov5", --> It should be yolov5 for YoloV5 model.

          "model-path":"../models/yolov5.dlc", --> Path of the quantized model

          "label-path":"../data/label.txt", --> Path to the label file

          "runtime":"DSP",

          "labels":85, --> Update label here. it should be 5 + {number of classes}. If nc = 3 then labels = 8

          "grids":25200,

          "nms-threshold":0.5,

          "conf-threshold":0.4,

          "input-layers":[

              "images" --> Open the model in netron.app and get the input-layers names.

],

          "output-layers":[ --> Refer the steps given above to know the output-layers and output-tensors

              "Conv_271",

              "Conv_305",

              "Conv_339"

],

          "output-tensors":[

              "443",

              "496",

              "549"

],

          "global-threshold":0.2

},

4. 在目标设备上下载和部署SNPE库的步骤

Download the SDK from Qualcomm Neural Processing SDK for AI Tools & Resources Archive - Qualcomm Developer Network

Windows

cd snpe-1.68.0\snpe-1.68.0.3932

adb push lib\aarch64-ubuntu-gcc7.5\. /usr/lib/

adb push lib\aarch64-ubuntu-gcc7.5\libsnpe_dsp_domains_v2.so /usr/lib/rfsa/adsp/

adb push lib\dsp\. /usr/lib/rfsa/adsp/

adb push bin\aarch64-ubuntu-gcc7.5\snpe-net-run /usr/bin/

Linux

cd snpe-1.68.0/snpe-1.68.0.3932/

adb push lib/aarch64-ubuntu-gcc7.5/* /usr/lib/

adb push lib/aarch64-ubuntu-gcc7.5/libsnpe_dsp_domains_v2.so /usr/lib/rfsa/adsp/

adb push lib/dsp/* /usr/lib/rfsa/adsp/

adb push bin/aarch64-ubuntu-gcc7.5/snpe-net-run /usr/bin/

Verify SDK version

adb shell

chmod +x /usr/bin/snpe-net-run

snpe-net-run --version

5. 执行应用程序的步骤

5.1 安装 OpenCV 4.5.5

下载OpenCV 4.5.5源代码

https://codeload.github.com/opencv/opencv/tar.gz/refs/tags/4.5.5

adb shell

wget https://codeload.github.com/opencv/opencv/tar.gz/refs/tags/4.5.5 -O opencv-4.5.5.tar.gz

tar  -zxvf opencv-4.5.5.tar.gz

cd ./opencv-4.5.5

安装依赖项

apt install build-essential cmake unzip git pkg-config

apt install libjpeg-dev libpng-dev libtiff-dev

apt-get install libjsoncpp-dev libjson-glib-dev libgflags-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio

apt install libjasper-dev

apt-get install libeigen3-dev

如果你收到一个关于libjasper-dev丢失的错误，请按照以下说明进行操作：

wget http://ports.ubuntu.com/ubuntu-ports/pool/main/j/jasper/libjasper-dev_1.900.1-debian1-2.4ubuntu1.3_arm64.deb

dpkg -i libjasper-dev_1.900.1-debian1-2.4ubuntu1.3_arm64.deb

wget http://ports.ubuntu.com/ubuntu-ports/pool/main/j/jasper/libjasper1_1.900.1-debian1-2.4ubuntu1.3_arm64.deb

dpkg -i libjasper1_1.900.1-debian1-2.4ubuntu1.3_arm64.deb

否则(如果已经安装libjasper-dev), 继续执行：

apt install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev

apt install libxvidcore-dev libx264-dev

OpenCV的highgui模块依赖于GTK库进行GUI操作。安装GTK命令：

apt install libgtk-3-dev

安装Python 3头文件和库

apt install libatlas-base-dev gfortran

apt install python3.6-dev

Build and install

mkdir build && cd build

cmake -D CMAKE_BUILD_TYPE=RELEASE \

      -D CMAKE_INSTALL_PREFIX=/usr/local/opencv4.5 \

      -D OPENCV_ENABLE_NONFREE=ON \

      -D OPENCV_GENERATE_PKGCONFIG=YES \

      -D WITH_QT=ON \

      -D WITH_OPENGL=ON \

      -D BUILD_EXAMPLES=OFF \

      -D INSTALL_PYTHON_EXAMPLES=OFF \

..

make -j8

make install

5.2 构建应用程序

将下载的SDK从主机推送到RB5

adb push snpe-1.68.0.zip /home/

adb shell

cd /home/

unzip snpe-1.68.0.zip

编译此应用程序

cd <Application Directory>

mkdir build

cd build

cmake -DSNPE_SDK_BASE_DIR=<SDK Directory Path>/snpe-1.68.0.3932 ..

make

5.3 运行此应用程序

在监视器上显示输出。请通过HDMI电缆将显示器连接到设备。按照以下说明启用weston：

export XDG_RUNTIME_DIR="/run/user/root"

cd build

./out/main -c ../data/config.json

验证结果

如果输出类型为filesink，请检查“输出路径”的目录是否为filesink。或者，请检查与HDMI连接的显示器的输出。

作者：廖洋洋，高通工程师

csdnsqst0050

关注

26
点赞
踩
22

收藏

觉得还不错? 一键收藏
1
评论
基于Qualcomm Robotics RB5边缘侧AI应用—人脸/口罩/人员计数目标检测

目标检测是机器视觉的常用的任务，本文介绍基于高通Snapdragon 与 Yolov5 模型实现边缘计算的场景实现人脸、口罩、人员计数。相关的代码已开源，可以在GitHub上获取。
复制链接

扫一扫