RK3588 NPU研究（一）

1. 使用NPU进行AI计算，具体对应模型推理

2. 对应代码位置
   hardware/rockchip/rknpu2

3. examples: yolo5 编译

   • 下载NDK工具https://developer.android.google.cn/ndk/downloads?hl=zh-cn
   • 源码中使用16b，测试可以使用16/17/18/19/20，不可以使用21，后续未测试
   • 修改：ANDROID_NDK_PATH=~/opt/android-ndk-r20b
   • 命令：bash build-android_RK3588.sh
   • 生成物：build目录下rknn_yolov5_demo，model/，lib/

4. examples: yolo5 运行

   post process config: box_conf_threshold = 0.25, nms_threshold = 0.45
   Read meetting03.jpg ...
   img width = 3840, img height = 2160
   Loading mode...
   sdk version: 1.5.2 (c6b7b351a@2023-08-23T15:27:35) driver version: 0.9.2
   model input num: 1, output num: 3
     index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, w_stride = 640, size_with_stride=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
     index=0, name=output, n_dims=4, dims=[1, 255, 80, 80], n_elems=1632000, size=1632000, w_stride = 0, size_with_stride=1638400, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003860
     index=1, name=283, n_dims=4, dims=[1, 255, 40, 40], n_elems=408000, size=408000, w_stride = 0, size_with_stride=491520, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
     index=2, name=285, n_dims=4, dims=[1, 255, 20, 20], n_elems=102000, size=102000, w_stride = 0, size_with_stride=163840, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003915
   model is NHWC input fmt
   model input height=640, width=640, channel=3
   resize with RGA!
   scal once run use 22.702000 ms
   once run use 20.701000 ms
   loadLabelName ./model/coco_80_labels_list.txt
   person @ (1482 475 1944 938) 0.704657
   person @ (792 604 1632 2116) 0.614134
   person @ (126 661 882 1815) 0.602546
   person @ (84 540 432 1096) 0.582360
   person @ (1290 428 1644 860) 0.570635
   person @ (1272 759 2784 2160) 0.494397
   person @ (2952 475 3834 1383) 0.484189
   person @ (1800 506 2340 968) 0.442070
   loop count = 1000 , average run  21.540229 ms
   emo model/RK3588/yolov5s-640-640.rknn meetting03.jpg                          <
   post process config: box_conf_threshold = 0.25, nms_threshold = 0.45
   Read meetting03.jpg ...
   img width = 1920, img height = 1080
   Loading mode...
   sdk version: 1.5.2 (c6b7b351a@2023-08-23T15:27:35) driver version: 0.9.2
   model input num: 1, output num: 3
     index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, w_stride = 640, size_with_stride=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
     index=0, name=output, n_dims=4, dims=[1, 255, 80, 80], n_elems=1632000, size=1632000, w_stride = 0, size_with_stride=1638400, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003860
     index=1, name=283, n_dims=4, dims=[1, 255, 40, 40], n_elems=408000, size=408000, w_stride = 0, size_with_stride=491520, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
     index=2, name=285, n_dims=4, dims=[1, 255, 20, 20], n_elems=102000, size=102000, w_stride = 0, size_with_stride=163840, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003915
   model is NHWC input fmt
   model input height=640, width=640, channel=3
   resize with RGA!
   scal once run use 13.794000 ms
   once run use 21.535000 ms
   loadLabelName ./model/coco_80_labels_list.txt
   person @ (738 237 975 469) 0.704657
   person @ (396 303 816 1051) 0.614134
   person @ (57 327 444 897) 0.604937
   person @ (645 214 822 426) 0.601903
   person @ (42 270 216 551) 0.566727
   person @ (645 386 1389 1080) 0.486672
   person @ (1473 237 1911 691) 0.472939
   person @ (903 253 1173 482) 0.426558
   loop count = 1000 , average run  21.586659 ms
   

   

5. examples: yolo5 分析

   图像数据流程

   • model的输入格式是RGB888，大小是640x640，输入需要被识别文件为jpg，用opencv进行转换缩放

   • cv::Mat orig_img = cv::imread(image_name, 1);
   • cv::cvtColor(orig_img, img, cv::COLOR_BGR2RGB);
   • src = wrapbuffer_virtualaddr((void*)img.data, img_width, img_height, RK_FORMAT_RGB_888);
   • dst = wrapbuffer_virtualaddr((void*)resize_buf, width, height, RK_FORMAT_RGB_888);
   • IM_STATUS STATUS = imresize(src, dst);
   • inputs[0].buf = resize_buf;

   • 图像缩放使用的rga，4k rgb需要12ms，4k yuv需要9ms

   model运行流程

   • 加载模型、rknn初始化、获取rknn信息、执行rknn，获取结果、后期处理、结果释放、后期处理资源释放、rknn资源释放

   • unsigned char* model_data = load_model(model_name, &model_data_size);
   • ret = rknn_init(&ctx, model_data, model_data_size, 0, NULL);
   • ret = rknn_query(ctx, RKNN_QUERY_SDK_VERSION, &version, sizeof(rknn_sdk_version));
   • ret = rknn_query(ctx, RKNN_QUERY_IN_OUT_NUM, &io_num, sizeof(io_num));
   • ret = rknn_query(ctx, RKNN_QUERY_INPUT_ATTR, &(input_attrs[i]), sizeof(rknn_tensor_attr));
   • ret = rknn_query(ctx, RKNN_QUERY_OUTPUT_ATTR, &(output_attrs[i]), sizeof(rknn_tensor_attr));
   • ret = rknn_run(ctx, NULL);
   • ret = rknn_outputs_get(ctx, io_num.n_output, outputs, NULL);
   • post_process((int8_t*)outputs[0].buf, (int8_t*)outputs[1].buf, (int8_t*)outputs[2].buf,…);
   • ret = rknn_outputs_release(ctx, io_num.n_output, outputs);
   • deinitPostProcess();
   • ret = rknn_destroy(ctx);