XIAO ESP32S3之运行物体检测例程

一、例程介绍 

         此例程是运行FOMO 轻量检测模型实现物体检测，Demo中已包含训练好的模型参数，无需再训练。

        FOMO(Faster Objects, More Objects) 是由 Edgeimpulse 工程师提出的一种轻量级的目标检测模型，其主要特点是模型非常小，计算量也很小，表现出来就是速度非常快，精度也不错。FOMO 的优点是速度快，缺点是精度不高，但是在一些对精度要求不高的场景下，FOMO 是一个不错的选择。

FOMO介绍：FOMO 轻量检测模型 - Sipeed Wiki     

二、程序讲解

1、代码位置

\edgelab-example-esp32\examples\fomo_detection_demo

编程语言C++。

2、文件说明 

app_main.cpp        //主文件，加载模型，获取图像，算法检测，打印结果

fomo_mobilenetv2_model_data.cpp        //模型参数文件

fomo_mobilenetv2_model_data.h           //模型参数头文件

3、入口函数分析

extern "C" void app_main(void) {
    using namespace edgelab;    //使用edgelab。

    Device*  device  = Device::get_device();    //创建一个Device对象，Device类在\edgelab-example-esp32\components\sscma-micro\porting\el_device.h文件中定义。
    Display* display = device->get_display();    //定义一个显示器变量并赋值，但XIAO ESP32S3套件没有显示屏，需要去掉。
    Camera*  camera  = device->get_camera();    //定义一个相机变量并赋值。Camera类在\edgelab-example-esp32\components\sscma-micro\porting\el_camera.h文件中定义。

    display->init();    //显示屏初始化，但XIAO ESP32S3套件没有显示屏，需要去掉。
    camera->init(240, 240);    //摄像头初始化。

    auto* engine       = new EngineTFLite();    //创建TFLite模型引擎。
    auto* tensor_arena = heap_caps_malloc(kTensorArenaSize, MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT);    //从SPIRAM中分配kTensorArenaSize (1024 * 1024)个8位宽的内存。
    engine->init(tensor_arena, kTensorArenaSize);    //初始化引擎。
    engine->load_model(g_fomo_mobilenetv2_model_data, g_fomo_mobilenetv2_model_data_len);    //引擎加载模型参数。
    auto* algorithm = new AlgorithmFOMO(engine);    //创建一个FOMO模型算法对象。

    while (true) {
        el_img_t img;    //定义图像变量。
        camera->start_stream();    //启动摄像头拍摄视频流。
        camera->get_frame(&img);    //获取一帧图像数据。
        algorithm->run(&img);        //运行算法检测图像中是否有物体。
        uint32_t preprocess_time  = algorithm->get_preprocess_time();
        uint32_t run_time         = algorithm->get_run_time();
        uint32_t postprocess_time = algorithm->get_postprocess_time();
        uint8_t  i                = 0u;
        for (const auto& box : algorithm->get_results()) {
            el_printf("\tbox -> cx_cy_w_h: [%d, %d, %d, %d] t: [%d] s: [%d]\n",
                      box.x,
                      box.y,
                      box.w,
                      box.h,
                      box.target,
                      box.score);

            int16_t y = box.y - box.h / 2;
            int16_t x = box.x - box.w / 2;
            el_draw_rect(&img, x, y, box.w, box.h, color[++i % 5], 4);    //在图片上加入标识框。
        }
        el_printf("preprocess: %d, run: %d, postprocess: %d\n", preprocess_time, run_time, postprocess_time);      
        display->show(&img);      //显示屏显示图像，但XIAO ESP32S3套件没有显示屏，需要去掉。
        camera->stop_stream();    //停止相机拍摄视频流。
    }

    delete algorithm;
    delete engine;
}

三、代码编译

1、去掉display

先去掉app_main()中display相关的代码。

2、加入device-init()

在Device*  device  = Device::get_device();之后加入以下两行代码：

    device->init();
    printf("device_name:%s\n", device->get_device_name() );

这是Demo的一个bug，如果不加入，程序运行会出现内存错误，ESP32S3会重启。 

3、编译

执行以下指令：

cd fomo_detection_demo

idf.py set-target esp32s3

idf.py build

编译成功后日志如下： 

Creating esp32s3 image...
Merged 2 ELF sections
Successfully created esp32s3 image.
Generated F:/esp/esp-idf/projects/edgelab-example-esp32/examples/fomo_detection_demo/build/fomo_detection_demo.bin
[1263/1263] cmd.exe /C "cd /D F:\esp\esp-idf\projects\edge...xamples/fomo_detection_demo/build/fomo_detection_demo.bin"
fomo_detection_demo.bin binary size 0x1327a0 bytes. Smallest app partition is 0x3bc000 bytes. 0x289860 bytes (68%) free.

Project build complete. To flash, run:
 idf.py flash
or
 idf.py -p PORT flash
or
 python -m esptool --chip esp32s3 -b 460800 --before default_reset --after hard_reset --no-stub write_flash --flash_mode dio --flash_size 8MB --flash_freq 80m 0x0 build\bootloader\bootloader.bin 0x8000 build\partition_table\partition-table.bin 0x10000 build\fomo_detection_demo.bin
or from the "F:\esp\esp-idf\projects\edgelab-example-esp32\examples\fomo_detection_demo\build" directory
 python -m esptool --chip esp32s3 -b 460800 --before default_reset --after hard_reset --no-stub write_flash "@flash_args"

四、烧录

1、开发板进入BOOT模式

执行idf.py flash命令，开发板会自动进入boot模式。

2、烧写固件

使用idf.py flash或idf.py -p PORT flash指令进行烧录。第一个命令没有指定COM编号，如果电脑当前只有一个COM可以用此指令。如果电脑接了多个COM，需要用第二条指令，其中PORT为COMx，如idf.py -p COM5 flash。

五、运行

执行idf.py monitor指令开启运行监视功能。

运行日志：

I (303) esp_image: segment 3: paddr=00129238 vaddr=3fca19I (738) esp_psram: SPI SRAM memory test OK
I (747) cpu_start: Pro cpu start user code
I (747) cpu_start: cpu freq: 240000000 Hz
I (747) cpu_start: Application information:
I (747) cpu_start: Project name:     fomo_detection_demo
I (747) cpu_start: App version:      040a7b4-dirty
I (748) cpu_start: Compile time:     Dec 17 2023 00:42:49
I (748) cpu_start: ELF file SHA256:  08b96bf32...
I (748) cpu_start: ESP-IDF:          v5.3-dev-741-g03414a1550-dirty
I (748) cpu_start: Min chip rev:     v0.0
I (749) cpu_start: Max chip rev:     v0.99
I (749) cpu_start: Chip rev:         v0.1
I (749) heap_init: Initializing. RAM available for dynamic allocation:
I (749) heap_init: At 3FCA7A98 len 00041C78 (263 KiB): RAM
I (749) heap_init: At 3FCE9710 len 00005724 (21 KiB): RAM
I (750) heap_init: At 600FE010 len 00001FD8 (7 KiB): RTCRAM
I (750) esp_psram: Adding pool of 8192K of PSRAM memory to heap allocator
I (751) spi_flash: detected chip: gd
I (751) spi_flash: flash io: qio
W (751) i2c: This driver is an old driver, please migrate your application code to adapt `driver/i2c_master.h`
I (752) sleep: Configure to isolate all GPIO pins in sleep state
I (752) sleep: Enable automatic switching of GPIO sleep configuration
I (753) main_task: Started on CPU0
I (763) esp_psram: Reserving pool of 32K of internal memory for DMA/internal allocations
I (763) main_task: Calling app_main()
device_name:Seeed Studio XIAO (ESP32-S3)
I (763) s3 ll_cam: DMA Channel=4
I (763) cam_hal: cam init ok
I (763) sccb: pin_sda 40 pin_scl 39
I (763) sccb: sccb_i2c_port=1
I (773) camera: Detected camera at address=0x30
I (773) camera: Detected OV2640 camera
I (773) camera: Camera PID=0x26 VER=0x42 MIDL=0x7f MIDH=0xa2
I (843) s3 ll_cam: node_size: 3840, nodes_per_line: 1, lines_per_node: 8
I (843) s3 ll_cam: dma_half_buffer_min:  3840, dma_half_buffer: 11520, lines_per_half_buffer: 24, dma_buffer_size: 23040
I (843) cam_hal: buffer_size: 23040, half_buffer_size: 11520, node_buffer_size: 3840, node_cnt: 6, total_cnt: 10
I (843) cam_hal: Allocating 115200 Byte frame buffer in PSRAM
I (843) cam_hal: cam config ok
I (843) ov2640: Set PLL: clk_2x: 1, clk_div: 3, pclk_auto: 1, pclk_div: 8
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 3, run: 51, postprocess: 1
preprocess: 3, run: 51, postprocess: 1
preprocess: 3, run: 51, postprocess: 0
preprocess: 3, run: 51, postprocess: 0
preprocess: 3, run: 51, postprocess: 0
preprocess: 3, run: 51, postprocess: 0
preprocess: 3, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 4, run: 51, postprocess: 0
preprocess: 3, run: 51, postprocess: 1
box -> cx_cy_w_h: [90, 190, 20, 20] t: [1] s: [85]

 日志""打印"box -> cx_cy_w_h:"表示检测到了物体，后面的数值是物体的区域。

六、遇到问题

Waiting for the device to reconnect.
I (303) esp_image: segment 3: paddr=00129220 vaddr=3fca19I (738) esp_psram: SPI SRAM memory test OK
I (747) cpu_start: Pro cpu start user code
I (747) cpu_start: cpu freq: 240000000 Hz
I (747) cpu_start: Application information:
I (747) cpu_start: Project name:     fomo_detection_demo
I (747) cpu_start: App version:      040a7b4-dirty
I (748) cpu_start: Compile time:     Dec 17 2023 00:42:49
I (748) cpu_start: ELF file SHA256:  95d9277c7...
I (748) cpu_start: ESP-IDF:          v5.3-dev-741-g03414a1550-dirty
I (748) cpu_start: Min chip rev:     v0.0
I (749) cpu_start: Max chip rev:     v0.99
I (749) cpu_start: Chip rev:         v0.1
I (749) heap_init: Initializing. RAM available for dynamic allocation:
I (749) heap_init: At 3FCA7A98 len 00041C78 (263 KiB): RAM
I (749) heap_init: At 3FCE9710 len 00005724 (21 KiB): RAM
I (750) heap_init: At 600FE010 len 00001FD8 (7 KiB): RTCRAM
I (750) esp_psram: Adding pool of 8192K of PSRAM memory to heap allocator
I (751) spi_flash: detected chip: gd
I (751) spi_flash: flash io: qio
W (751) i2c: This driver is an old driver, please migrate your application code to adapt `driver/i2c_master.h`
I (751) sleep: Configure to isolate all GPIO pins in sleep state
I (752) sleep: Enable automatic switching of GPIO sleep configuration
I (752) main_task: Started on CPU0
I (762) esp_psram: Reserving pool of 32K of internal memory for DMA/internal allocations
I (762) main_task: Calling app_main()
Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.

Core  0 register dump:
PC      : 0x42009618  PS      : 0x00060c30  A0      : 0x820d815e  A1      : 0x3fcab5c0
0x42009618: app_main at F:/esp/esp-idf/projects/edgelab-example-esp32/examples/fomo_detection_demo/main/app_main.cpp:27

A2      : 0x00000003  A3      : 0x3c0e405c  A4      : 0x00000000  A5      : 0x000002fa
A6      : 0x3c0e405c  A7      : 0x3fcab620  A8      : 0x82009616  A9      : 0x3fcab5a0
A10     : 0x3fca3d80  A11     : 0x3c0e405c  A12     : 0x3c0e4128  A13     : 0x3fcab600
A14     : 0x3fcab5e0  A15     : 0x3fca3d80  SAR     : 0x0000000c  EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000000  LBEG    : 0x400570e8  LEND    : 0x400570f3  LCOUNT  : 0x00000000
0x400570e8: memset in ROM

0x400570f3: memset in ROM

从以上日志可以看出执行camera->init(240, 240);代码让ESP32重启了。

warning: unknown kconfig symbol 'CAMERA_MODULE_ESP_S3_EYE' assigned to 'y' in F:/esp/esp-idf/projects/edgelab-example-esp32/examples/fomo_detection_demo/sdkconfig.defaults.esp32s3

解决方法：参看3.2节。