该解决方案将使用IP摄像头、基于高通®Robotics RB5 Edge AI设备的实时输入流分析和计算机视觉系统。解决方案提供了一个端到端的应用程序,可以快速集成到新的构建系统中。相关代码可以在这里获取到:
所需使用硬件设备
- An Ubuntu 20.04 PC
- Qualcomm Robotics RB5 Development kit: https://developer.qualcomm.com/qualcomm-robotics-rb5-kit
- A USB camera
- A display monitor
相关环境配置
1. 前提条件
- One x86 Host Workstation. Operating System Ubuntu 20.04, Python 3.6.9
- Qualcomm Robotics RB5 Development kit: https://developer.qualcomm.com/qualcomm-robotics-rb5-kit
2. x86 Host Installation and Deep Learning Model Compile
应用程序中使用的模型需要使用数据集进行训练。然后将模型转换为dlc以在应用程序中运行。
2.1 Qualcomm® Neural Processing SDK Installation
Download Qualcomm® Neural Processing Software Development Kit (SDK) from Qualcomm Neural Processing SDK for AI Tools & Resources Archive - Qualcomm Developer Network.
高通神经处理SDK提供了用于模型转换(onnx到dlc)、模型量化和执行的工具。请参阅SDK中详细文档中给出的步骤进行安装. Snapdragon Neural Processing Engine SDK: Main Page
2.2 准备Yolov5 AI模型
按照 ModelPreperation.md 导出和编译Yolov5模型。
下载人员检测预训练YoloV5模型https://github.com/ultralytics/yolov5/releases/download/v6.0/yolov5s.pt
对于头盔和头部检测,需要在数据集上训练模型。头盔和头部检测数据集可以从Kaggle下载。
Safety Helmet Detection | Kaggle
2.3 Export *.pt file to ONNX file
建议安装Python 3.6.9版本。下载Yolov5 v6.0源代码
wget https://github.com/ultralytics/yolov5/archive/refs/tags/v6.0.tar.gz
tar -zxvf v6.0.tar.gz
cd yolov5-6.0
wget https://github.com/ultralytics/yolov5/releases/download/v6.0/yolov5s.pt
Python要求:
- python 3.6.9
- protobuf==3.17.3
- onnx==1.9.0
- onnx-simplifier==0.2.6
- onnxoptimizer==0.2.6
- onnxruntime==1.1.0
- numpy==1.16.5
- torch==1.10.0
- torchvision==0.11.1
sudo apt install python3.6 python3.6-venv build-essential make python3-dev python3.6-dev protobuf-compiler libprotobuf-dev
python3.6 -m pip install --upgrade pip
python3.6 -m pip install -r requirements.txt
python3.6 -m pip install coremltools>=4.1 onnx==1.9.0 scikit-learn==0.19.2 onnxruntime==1.1.0 onnx-simplifier==0.2.6 onnxoptimizer==0.2.6
将YoloV5导出到ONNX:
python3.6 export.py --weights yolov5s.pt --optimize --opset 11 --simplify --include onnx -imgsz [416,416]
3. 将Onnx模型转换为DLC
3.1 设置SNPE
需要设置SNPE将onnx模型转换为量化的dlc,请使用提供的链接按照说明设置神经处理SDK。请在整个演示过程中使用相同版本的SNPE. https://developer.qualcomm.com/docs/snpe/setup.html
export SNPE_ROOT=<path to snpe folder>/snpe-1.68.0.3932
export ANDROID_NDK_ROOT=<path to android ndk folder>
从pip查找onnx安装路径
python3.6 -m pip show onnx
在上述命令的输出中查找“Location:”行
export ONNX_DIR=<path from Location line>/onnx
为snpe设置onnx环境
cd $SNPE_ROOT
source bin/envsetup.sh -o $ONNX_DIR
SNPE目前不支持5D运算符。它要求在转换命令中的5D重塑之前指定输出节点。可以在Netron中检查输出节点。
要检查输出层节点,请在Netron应用程序中打开模型,然后单击Conv层。
在附加的快照中,5D之前的输出节点为onnx::326(Conv_198)、365(Conv_216)和404(Conv_334)
此实现具有以下功能:
- anchorBoxProcess: Get raw data from out nodes before 5D (Conv_198, Conv_216, Conv_234), convert to meaning data (scores, class, bounding boxes).
- doNMS: (non-max suppression): remove overlap boxes
- ShowDetectionOverlay: Overlay detection result at output video/Image
3.2 转换为DLC
snpe-onnx-to-dlc -i yolov5s.onnx --out_node <output name 1> --out_node <output name 2> --out_node <output name 3>
与上面的屏幕截图相对应的示例:
snpe-onnx-to-dlc -i yolov5s.onnx --out_node 326 --out_node 365 --out_node 404
3.3 生成AIP/DSP的量化模型
使用设备中安装的相同高通®神经处理SDK版本。请参考模型目录中给出的inputlist.txt文件。根据您的yolov5s.onnx模型创建您自己的inputlist.txt文件。我们需要更新inputlist.txt中的所有输出名称
snpe-dlc-quantize --input_dlc=yolov5s.dlc --input_list=inputlist.txt --output_dlc=yolov5s_quant.dlc --enable_hta
模型量化需要input.raw文件。使用以下python命令创建一个示例input.raw文件。
import numpy as np
((np.random.random((1,3,640,640)).astype(np.float32))).tofile("input.raw")
- Please read Snapdragon Neural Processing Engine SDK: Quantized vs Non-Quantized Models to know more about model quantization.
4. X86 Host RTSP Stream Installation
在本节中,演示如何准备测试视频并设置live555 RTSP流媒体服务器以模拟IP摄像机。安装步骤在X86主机中运行。
4.1 准备测试视频
如果测试视频是mp4、mkv或其他格式,请准备一个测试视频。需要转换为H264原始视频。以下步骤演示如何将MP4转换为H264原始视频。
wget https://test-videos.co.uk/vids/bigbuckbunny/mp4/h264/1080/Big_Buck_Bunny_1080_10s_1MB.mp4
sudo apt install ffmpeg
ffmpeg -i Big_Buck_Bunny_1080_10s_1MB.mp4 -f h264 -vcodec libx264 Big_Buck_Bunny_1080_10s_1MB.264
4.2 Live555服务器安装
wget http://www.live555.com/liveMedia/public/live555-latest.tar.gz
tar -zxvf live555-latest.tar.gz
cd live/
./genMakefiles linux-64bit
make -j4
cd ..
将测试视频复制到mediaServer文件夹
cp Big_Buck_Bunny_1080_10s_1MB.264 ./live/mediaServer
cd ./live/mediaServer
./live555MediaServer
rtsp://192.168.4.111:8554/ is the rtsp url.
- "192.168.4.111" is RTSP server IP address
- 8554 is the default port
- is the video file name under mediaServer folder In this case, url "rtsp://192.168.4.111:8554/Big_Buck_Bunny_1080_10s_1MB.264" is the video address.
4.3 验证live555服务器
在Windows桌面上下载并安装VLC媒体播放器VLC: Official site - Free multimedia solutions for all OS! - VideoLAN
Launch VLC player, choose "Media->Open Network Stream" input RTSP url rtsp://192.168.4.111:8554/Big_Buck_Bunny_1080_10s_1MB.264
单击播放以测试live555服务器是否正常工作。注意:确保网络地址可访问
如何配置、构建和运行此人群热力检测应用程序
1. 克隆存储库
adb shell
cd /home/
apt install git
git clone https://github.com/quic/sample-apps-for-robotics-platforms.git
cd sample-apps-for-robotics-platforms/RB5/linux_kernel_5_x/AI-ML-apps/AI_Heatmap_Solutions/
2. 更新应用程序配置
所有解决方案的配置在data/config.json文件中进行了描述。应更新此配置文件以选择所需的解决方案、模型配置和输入/输出流。应用程序可以将rtsp/camera流作为输入,并将输出转储到mp4或hdmi监视器。
Table 1-1 show all the configuration items:
输入配置
key | Value | Description |
input-config-name | string | Name of the input config |
stream-type | string | Input stream type camera or rtsp |
stream-width | int | Width of the input stream |
stream-height | int | Height of the input stream |
SkipFrame | int | Numer of frames to skip |
camera-url | string | rtsp stream path if the input stream is rtsp |
模型配置
key | Value | Description |
model-name | string | Name of the model |
model-path | string | Path of the dlc file |
label-path | string | Path of the label file |
runtime | string | SNPE Runtime (GPU, CPU, DSP) |
nms-threshold | float | NMS threshold |
conf-threshold | float | Confidence threshold |
labels | int | Number of labels + 5 |
input-layers | string | Name of the input layers |
output-layers | string | Name of the output layers |
output-tensors | string | Name of the output tensors |
解决方案配置
key | Value | Description |
solution-name | string | Name of the Solution |
model-name | string | Name of the model configuration to be used |
input-config-name | string | Name of the Input configuration to be used |
Enable | bool | 1 to Enable and 0 to Disable the solution |
output-type | string | Filesink to save the output in mp4 |
Wayland if display the output on hdmi monitor | ||
output-path | string | Path of the output, Enabled if output type |
is filesink |
Example 1: 从相机获取输入流并在hdmi监视器上输出的配置
{
"input-configs":[
{
"input-config-name":"camera",
"stream-type":"camera",
"stream-width":1280,
"stream-height":720,
"SkipFrame":1,
"fps-n":30,
"fps-d":1
},
],
"model-configs":[
{
"model-name":"yolov5s-1",
"model-path":"../models/heatmap_detection.dlc",
"label-path":"../data/coco_label.txt",
"runtime":"DSP",
"labels":85,
"grids":25200,
"nms-threshold":0.5,
"conf-threshold":0.5,
"input-layers":[
"images"
],
"output-layers":[
"Conv_271",
"Conv_305",
"Conv_339"
],
"output-tensors":[
"443",
"496",
"549"
],
"global-threshold":0.2
},
],
"solution-configs":[
{
"solution-name":"heatmap-detection",
"model-name":"yolov5s-1",
"input-config-name":"camera",
"Enable":0,
"strict_area_x":10,
"strict_area_y":10,
"strict_area_w":1000,
"strict_area_h":1000,
"output-type":"wayland",
},
],
}
Example 2: 设备上rtsp输入流和输出的配置
"input-configs":[
{
"input-config-name":"rtsp3",
"stream-type":"rtsp",
"camera-url":"rtsp://10.147.243.253:8554/heatmap_video.264",
"SkipFrame":1
},
"model-configs":[
{
"model-name":"yolov5s-3",
"model-path":"../models/heatmap_detection.dlc",
"label-path":"../data/coco_labels.txt",
"runtime":"DSP",
"nms-threshold":0.1,
"conf-threshold":0.1,
"labels":6,
"grids":25200,
"input-layers":[
"images"
],
"output-layers":[
"/model.24/m.0/Conv",
"/model.24/m.1/Conv",
"/model.24/m.2/Conv"
],
"output-tensors":[
"/model.24/m.0/Conv_output_0",
"/model.24/m.1/Conv_output_0",
"/model.24/m.2/Conv_output_0"
],
"global-threshold":0.2
},
"solution-configs":[
{
"solution-name":"heatmap-solution",
"model-name":"yolov5s-3",
"input-config-name":"rtsp3",
"Enable":1,
"output-type":"filesink",
"output-path":"/root/crack.mp4"
},
]
Use model-name and input-config-name to select model and input stream respectively.
3. 模型集成
将model推送到应用程序中的model目录中,并更新config.json文件。
更新输出层和输出张量。要检查输出层和输出张量节点,请在Netron应用程序中打开模型,然后单击图像中提到的Conv层。
在yolov5s.onnx中,5D之前的输出层是onnx::326(Conv_198)、365(Conv_216)和404(Conv_34),输出张量是Conv_198Conv_216Conv_234。
"model-configs":[
{
"model-name":"yolov5s-1", --> Add model name here. It should match with the model name in solution config
"model-type":"yolov5", --> It should be yolov5 for YoloV5 model.
"model-path":"../models/yolov5.dlc", --> Path of the quantized model
"label-path":"../data/label.txt", --> Path to the label file
"runtime":"DSP",
"labels":8, --> Update label here. it should be 5 + {number of classes}. If nc = 3 then labels = 8
"grids":25200,
"nms-threshold":0.5,
"conf-threshold":0.4,
"input-layers":[
"images" --> Open the model in netron.app and get the input-layers names.
],
"output-layers":[ --> Refer the steps given above to know the output-layers and output-tensors
"Conv_198",
"Conv_216",
"Conv_234"
],
"output-tensors":[
"326",
"365",
"404"
],
"global-threshold":0.2
},
4. 在目标设备上下载和部署SNPE库的步骤
Download Qualcomm Neural Processing Software Development Kit (SDK) from Qualcomm Neural Processing SDK for AI Tools & Resources Archive - Qualcomm Developer Network.
Windows
cd snpe-1.68.0\snpe-1.68.0.3932
adb push lib\aarch64-ubuntu-gcc7.5\. /usr/lib/
adb push lib\aarch64-ubuntu-gcc7.5\libsnpe_dsp_domains_v2.so /usr/lib/rfsa/adsp/
adb push lib\dsp\. /usr/lib/rfsa/adsp/
adb push bin\aarch64-ubuntu-gcc7.5\snpe-net-run /usr/bin/
Linux
cd snpe-1.68.0/snpe-1.68.0.3932/
adb push lib/aarch64-ubuntu-gcc7.5/* /usr/lib/
adb push lib/aarch64-ubuntu-gcc7.5/libsnpe_dsp_domains_v2.so /usr/lib/rfsa/adsp/
adb push lib/dsp/* /usr/lib/rfsa/adsp/
adb push bin/aarch64-ubuntu-gcc7.5/snpe-net-run /usr/bin/
Verify SNPE version
adb shell
chmod +x /usr/bin/snpe-net-run
snpe-net-run --version
5. 执行应用程序的步骤
5.1 安装 OpenCV 4.5.5
下载OpenCV 4.5.5源代码
https://codeload.github.com/opencv/opencv/tar.gz/refs/tags/4.5.5
adb shell
wget https://codeload.github.com/opencv/opencv/tar.gz/refs/tags/4.5.5 -O opencv-4.5.5.tar.gz
tar -zxvf opencv-4.5.5.tar.gz
cd ./opencv-4.5.5
安装依赖项
apt install build-essential cmake unzip git pkg-config
apt install libjpeg-dev libpng-dev libtiff-dev
apt-get install libjsoncpp-dev libjson-glib-dev libgflags-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio
apt install libjasper-dev
apt-get install libeigen3-dev
如果你收到一个关于libjasper-dev丢失的错误,请按照以下说明进行操作:
wget http://ports.ubuntu.com/ubuntu-ports/pool/main/j/jasper/libjasper-dev_1.900.1-debian1-2.4ubuntu1.3_arm64.deb
dpkg -i libjasper-dev_1.900.1-debian1-2.4ubuntu1.3_arm64.deb
wget http://ports.ubuntu.com/ubuntu-ports/pool/main/j/jasper/libjasper1_1.900.1-debian1-2.4ubuntu1.3_arm64.deb
dpkg -i libjasper1_1.900.1-debian1-2.4ubuntu1.3_arm64.deb
否则(如果已经安装libjasper-dev), 继续执行:
apt install libavcodec-dev libavformat-dev libswscale-dev libv4l-dev
apt install libxvidcore-dev libx264-dev
OpenCV的highgui模块依赖于GTK库进行GUI操作。安装GTK命令:
apt install libgtk-3-dev
安装Python 3头文件和库
apt install libatlas-base-dev gfortran
apt install python3.6-dev
Build and install
mkdir build && cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local/opencv4.5 \
-D OPENCV_ENABLE_NONFREE=ON \
-D OPENCV_GENERATE_PKGCONFIG=YES \
-D WITH_QT=ON \
-D WITH_OPENGL=ON \
-D BUILD_EXAMPLES=OFF \
-D INSTALL_PYTHON_EXAMPLES=OFF \
..
make -j8
make install
5.2 构建应用程序
adb shell
cd /home/sample-apps-for-robotics-platforms/RB5/linux_kernel_5_x/AI-ML-apps/AI_Heatmap_Solution/
mkdir build
cd build
cmake -DSNPE_SDK_BASE_DIR=<SDK Directory Path>/snpe-1.68.0.3932 ..
make
5.3 运行应用程序
在监视器上显示输出。请通过HDMI电缆将显示器连接到设备。按照以下说明启用weston:
export XDG_RUNTIME_DIR=/run/user/root
cd build
./out/main -c ../data/config.json
验证结果
如果输出类型为filesink,请检查“输出路径”的目录是否为filesink。或者,请检查与HDMI连接的显示器的输出。
作者:廖洋洋,高通工程师