大疆行业无人机接入音视频平台协议详解

前言

从事大疆行业应用开发有一段时间了，看到很多厂商在做视频回传的时候，都要装个自己的APP，界面很丑不说，还经常卡死，但是大疆其实已经在视频流中携带了很多信息，很多人都不知道，现在把自己的直播开发经验分享出来

---

---

一、安防视频平台介绍

二、DJI Pilot机型匹配表及回传流程

APP 版本	DJI Pilot V1.9 及以上版本	DJI Pilot PE V1.6.1
适配机型	经纬 M300 RTK 经纬 M200 V2 系列 御 Mavic 2 行业变焦版 御 Mavic 2 行业双光版 精灵 Phantom 4 RTK（SDK 遥控器版本）	经纬 M200 V1 系列 经纬 M600 系列 悟 Inspire 2 御 Mavic Pro 精灵 Phantom 4 精灵 Phantom 4 Pro（ 除精灵 4 Pro+） 精灵 Phantom 4 Advanced（除精灵 4 Advanced+）

Pilot APP 下载：https://www.dji.com/cn/downloads/djiapp/dji-pilot
Pilot PE V1.6.1 下载：https://service-adhoc.dji.com/download/app/android/321b585b-e217-4c95-9192-09eee6e02630

1. 视频流选择

M300RTK 最多可以挂载 3 个相机负载，可以通过遥控器上的 APP 从三路相机视频流
中选择某一路视频流进行回传到服务端。

1. FPV 视频流：无人机前置摄像头第一视角视频流。
2. 1 号/2 号/3 号负载相机视频流：无人机上的 3 个云台口，可以任意挂载大疆的官方负载 H20T/ H20/ Z30/ XTS 相机，以及三方合作伙伴的相机。
3. H20T/H20 相机负载：多光组合相机，具备 1200 万像素广角、2000 万像素 23 倍光学变焦、1200 米激光测距、红外（H20T 有）。
4. Z30 相机负载：30 倍光学变焦相机。
5. XTS 相机负载：红外相机。

2. 视频回传的码流格式

1. 编码格式：H264。
2. 质量调节：流畅 540P 30FPS 512Kbps， 均衡 720P 30FPS 1Mbps， 高清 720P 30FPS 1.5 Mbps。
3. 传输格式：根据 GB/T28181-2016 传输要求,采用基于 RTP 的视频数据 PS 封装。

3. GB28181业务流程

4. 传输和控制要求（基于 GB/T 28181）

1. SIP 信令通信采用 UDP 协议，端口自定义，默认 5060。
2. RTP 数据通信采用 UDP 协议，多路流分别采用不同的端口分别传输，点播时通过SDP 协商，端口自定义。
3. RTCP 传输控制协议可选，默认 RTP 端口号加 1，需要收发双方同时支持才可，目前仅适用于网络质量反馈，暂不支持丢包重传机制。
4. 无人机视频编码格式为 H264，封装成 PS 流，再用 RTP 打包传输；不支持音频。
5. 接入网络应为 WiFi/有线/4G/VPDN 的一种，并在现场为遥控器提供 WiFi 热点。
6. 遥控器到服务器端的网络带宽应满足以下标准：流畅 512Kbps，均衡 1Mbps，高清1.5Mbps。
7. 接收端应具备 jitterbuffer 接收缓冲，以消除网络抖动和乱序造成的视频卡顿和花屏 ，建议缓冲不要超过 1 秒。

5. 功能列表

功能点	功能描述	协议接口	备注
设备注册注销	采用基于账号口令认证的设备登录与注销；采用定时心跳保活机制，在心跳超时离线后，自动重登录。
设备信息查询	支持远程查询无人机名称，ID，无人机型号，生产厂商，固件版本等信息
设备状态查询	支持远程查询无人机工作状态，视频流状态，录像状态，UTC时间等。
设备列表查询	支持远程查询摄像头列表，即具备多通道视频传输的能力。
实时视频点播与停止	设备上线后，采用点播即传的策略，按需上传，节省通信带宽和流量。
设备控制强制发送关键帧	当视频流丢帧或切换分辨率时，可通过设备控制信令，强制无人机立即编码发送一个 H264 关键帧，减少播放等待时间。

三、DJI Pilot H264 SEI帧 Metadata协议解析

目前无人机直播所用的 RTMP 协议只支持标准 RTMP 视频流及其内置的媒体控制命令，没有其他定制化
功能。

1. 编码格式：H.264，有 SEI 帧（SEI 帧主要是用于存放 Metadta 信息）
2. 传输格式：采用 RTMP 的 chunk 方式进行传输
3. 码流质量调节：
   • 不支持服务器进行质量设置调节。
   • 支持 APP 手动设置，参数如下：
     • 流畅 540P 30FPS 512Kbps
     • 均衡 720P 30FPS 1Mbps
     • 高清 720P 30FPS 1.5Mbps
4. 默认端口：默认使用 1935 端口，若服务器指定其他端口的话，需要先 PING 测试。
5. Metadata 数据项：
   • 飞机经纬高
   • 遥控器经纬高
   • 目标点位置信息

1. Metadata 数据列表

根据大疆内部文件列出，经实测有改动，具体以proto文件为准

类别	Metadata 名称	类型	单位	推送频率
飞机信息	飞机位置-经度	double	度	10Hz
飞机信息	飞机位置-纬度	double	度	10Hz
飞机信息	飞机位置-椭球高	float	米	10Hz
飞机信息	飞机姿态-roll	Int32	0.1 度	10Hz
飞机信息	飞机姿态-yaw	Int32	0.1 度	10Hz
飞机信息	飞机姿态-pitch	Int32	0.1 度	10Hz
飞机信息	Home 点-经度	Double	度	1Hz
飞机信息	Home 点-纬度	Double	度	1Hz
飞机信息	Home 点-椭球高	float	米	1Hz
遥控器信息	遥控器角色	enum		0.2Hz
遥控器信息	遥控器位置-经度	double	度	0.2Hz
遥控器信息	遥控器位置-纬度	double	度	0.2Hz
遥控器信息	遥控器位置-椭球高	float	米	0.2Hz
目标点	TargetPoint-版本	Uint32		2Hz
目标点	TargetPoint-Source	Char *		2Hz
目标点	TargetPoint-数据[数组]			2Hz

TargetPoint-数据[数组]

参数名	类型	单位
type	Enum
index	Uint32
latitude	Double	度
longitude	Double	度
altitude_ellipsoid	Float	米
altitude_relative	Float	米
video_stream_window_x	Float
video_stream_window_y	Float
flags	Uint32

2. DJI Metadata protobuf解析文件

DJI对Metadata采用protobuf进行封装，以下是proto文件，已经本人验证并修改了部分BUG

dvtm_library.proto

// DJI Video Timed Metadata Format using Protobuf 3

// version = 3

syntax = "proto3";
// basic message defin ==========================
enum DIRECTION_TYPE {
  RESERVED = 0;
  NORTH = 1;
  NORTHEAST = 2;
  EAST = 3;
  SOUTHEAST = 4;
  SOUTH = 5;
  SOUTHWEST = 6;
  WEST = 7;
  NORTHWEST = 8;
}

message DjiUTC {
  uint32 year = 1;       // The year, like 2020.
  uint32 month = 2;      // The month in the year, in the range 1 to 12.
  uint32 day = 3;        // The day of the month, in the range 1 to 31.
  uint32 hour = 4;       // The number of hours past midnight, in the range 0 to 23.
  uint32 min = 5;        // The number of minutes after the hour, in the range 0 to 59.
  uint32 sec = 6;        // The number of seconds after the minute, normally in the range 0 to 59.
  uint32 nsec = 7;       // The number of nanoseconds after the second.
}

message DeviceIdentify {
  string serial_number = 1;              // limited to 32 characters
}

message DjiInherentInfoBase {
  DeviceIdentify device_identify = 1;    // Include sn now.
  string         module_name = 2;        // limited to 32 characters
  string         firmware_version = 3;   // limited to 32 characters
  string         manufacture_id = 4;     // limited to 18 characters
  string         product_type = 5;       // limited to 32 characters
}

// [end] basic message defin ==========================

// Global Information, only appear once in whole video ==========================
message DjiModuleInfo {
  enum ModuleType {
    // The body of any kinds of camera which generate this video,
    // including gimbal camera, payload camera, or fly camera with
    // non-removable drone.
    MODULE_CAMERA_BODY = 0;

    // interchangeable drone
    MODULE_DRONE = 1;

    // interchangeable lens
    MODULE_LENS = 2;
  }
  ModuleType module_type = 1;
  string     model_name = 2;         // limited to 32 characters
  string     serial_number = 3;      // limited to 32 characters
  string     firmware_version = 4;   // limited to 32 characters
}

message DjiVideoGlobalInfo {
  repeated DjiModuleInfo module_info = 1;

  // The video full path file name, limited to 800 characters
  string  file_name = 3;

  fixed32 video_uuid = 4;             // the video unique ID
  string  record_start_time = 5;      // uses RFC 3339
  uint32  resolution_height = 6;      // unit: pixel
  uint32  resolution_width = 7;       // unit: pixel
  float   video_framerate = 8;
  enum VideoType {
    VIDEO_NORMAL = 0;
    VIDEO_DELAY = 1;
    VIDEO_SLOW_MOTION = 2;
    VIDEO_QUICK_MOVIE = 3;
    VIDEO_TIMELAPSE = 4;
    VIDEO_MOTIONLAPSE = 5;
    VIDEO_HYPERLAPSE = 6;
    VIDEO_HDR = 7;
    VIDEO_LOOP_RECORD = 8;
  }
  VideoType video_type = 9;
  enum VideoEncoder {
    ENCODER_H264 = 0;
    ENCODER_H265 = 1;
  }
  VideoEncoder video_encoder = 10;
  uint32 library_proto_version = 11; // version of dvtm_library.proto
  uint32 product_proto_version = 12; // version of product corresponding proto file
}
// [end] Global Information, only appear once in whole video ====================

// Describe every data source as a message ===================================

// All device_id field in messages is internal reserved at present.
// User should not care about or depends on this field.

message DjiCameraBasic {
  uint32 device_id = 1;
  int64  timestamp = 2;
  uint32 frame_id = 3;

  // reserved for future use
  string camera_name = 4;            // limited to 32 characters

  sint32 exposure_bias_tenfold = 5;  // exposure bias, tenfold expression, reserved for IR camera
  float  exposure_time = 6;          // exposure time (uint: s)
  uint32 iso = 7;                    // photographic sensitivity, reserved for IR camera
  uint32 fnumber_tenfold = 8;        // F-Number, tenfold expression
  float  focal_length = 9;           // actual focal length in mm
  float  digital_zoom_ratio = 10;
}

message DjiLaserRanging {
  uint32 device_id = 1;
  int64  timestamp = 2;
  uint32 frame_id = 3;

  // processed data of laser ranging function, fields 5-10 is only valid
  // if ranging_enabled is true, which means you enable this function
  // by yourself on APP. fields 6-8 is only valid if gps_status in DjiGpsBasic
  // message is not GPS_INVALID.
  bool   ranging_enabled = 4;
  uint32 target_distance = 5;        // mm
  float  target_longitude = 6;       // degree
  float  target_latitude = 7;        // degree
  uint32 target_altitude = 8;        // height relative to takeoff point in mm
  uint32 screen_offset_x = 9;        // target offset on horizontal direction of screen in permillage
  uint32 screen_offset_y = 10;       // target offset on vertical direction of screen in permillage

  // raw sensor data, always on. fields 12-19 is only valid if laser_status is LASER_NORMAL.
  enum LaserStatus {
    LASER_NORMAL = 0;                // laser ranging finder works fine.
    LASER_TOO_CLOSE = 1;             // target distance is less than minimum range of finder.
    LASER_TOO_FAR = 2;               // target distance is larger than maximum range of finder.
    LASER_CLOSED = 3;                // laser module is closed.
  }
  LaserStatus laser_status = 11;
  uint32 distance1 = 12;             // unit: mm
  uint32 intensity1 = 13;            // signal intensity, range: 0~255
  uint32 distance2 = 14;             // unit: mm
  uint32 intensity2 = 15;            // signal intensity, range: 0~255
  uint32 distance3 = 16;             // unit: mm
  uint32 intensity3 = 17;            // signal intensity, range: 0~255
  uint32 distance4 = 18;             // unit: mm
  uint32 intensity4 = 19;            // signal intensity, range: 0~255
  uint32 target_abs_alt = 20;        // absolute height of target point in mm
}

message DjiGpsBasic {
  uint32 device_id = 1;
  int64  timestamp = 2;
  uint32 frame_id = 3;
  double gps_latitude = 4;           // unit: rad, WGS-84 coordinate system
  double gps_longitude = 5;          // unit: rad, WGS-84 coordinate system
  int32  gps_altitude_mm = 6;        // unit mm, refer to gps_altitude_type for details
  enum GpsStatus {
    GPS_NORMAL = 0;                  // working with non-RTK GPS
    GPS_INVALID = 1;                 // GPS signal is non-available, measurement interrupted
    GPS_RTK = 2;                     // working with RTK-GPS
  }
  GpsStatus gps_status = 7;

  // Specify type of gps_altitude_mm and altitudes in other messages. For versions that don't support this fields,
  // use gps_status to get the type. If gps_status is GPS_RTK, gps_altitude_type is RTK_ALTITUDE, otherwise,
  // gps_altitude_type is PRESSURE_ALTITUDE.
  enum GpsAltType {
    PRESSURE_ALTITUDE = 0;           // altitude is mainly provided by barometer which has different origin with ellipsoidal height
    GPS_FUSION_ALTITUDE = 1;         // fused height by GPS and barometer, which based on ellipsoidal coordinate
    RTK_ALTITUDE = 2;                // altitude is ellipsoidal height provided by RTK
  }
  GpsAltType       gps_altitude_type = 8;
  enum COORDINATE_TYPE {
    WGS84 = 0;
    CGCS2000 = 1;
  }
  COORDINATE_TYPE  coordinate = 9;   // In rtk or gps mode, it indicate the coordinate of the lat, lon, alti.
  DjiUTC           utc_time = 10;
  DeviceIdentify   device_identify = 11;
}

message DjiFlyingState {
  uint32 device_id = 1;
  int64  timestamp = 2;
  uint32 frame_id = 3;

  // drone body speed components in NED coordinate system.
  int32  speed_x_dms = 4;            // north direction, unit: 0.1 m/s
  int32  speed_y_dms = 5;            // east direction, unit: 0.1 m/s
  int32  speed_z_dms = 6;            // vertical direction, unit: 0.1 m/s

  // The Euler angles of drone body relative to the NED (North, East, Down)
  // coordinate system. Rotation sequence of the Euler angle is
  // ZYX (yaw, pitch, roll), intrinsic.
  sint32 pitch_decidegree = 7;       // unit: 0.1 degree
  sint32 roll_decidegree = 8;        // unit: 0.1 degree
  sint32 yaw_decidegree = 9;         // unit: 0.1 degree

  // height relative to home point, unit: 0.1m
  sint32 relative_height_decimeter = 10;

  DjiUTC utc_time = 11;
  int32 heading = 12;                // unit: 0.1 degree
}

message DjiGimbal {
  uint32 device_id = 1;
  int64  timestamp = 2;
  uint32 frame_id = 3;
  enum GimbalPosition {
    GIMBAL_POS_NORMAL = 0;
    GIMBAL_POS_REVERSE = 1;
  }
  GimbalPosition gimbal_position = 4;
  enum GimbalMode {
    GIMBAL_MODE_FREE = 0;
    GIMBAL_MODE_FPV = 1;
    GIMBAL_MODE_FOLLOW = 2;
  }
  GimbalMode gimbal_mode = 5;

  // The Euler angles of gimbal relative to the NED (North, East, Down)
  // coordinate system. Rotation sequence of the Euler angle is
  // ZXY (yaw, roll, pitch), intrinsic. For upward gimbal, the Euler
  // angles translate from the real quaternion of gimbal after rotate
  // 180 degree around the X axis of moving body.
  sint32 pitch_decidegree = 6;       // unit: 0.1 degree
  sint32 roll_decidegree = 7;        // unit: 0.1 degree
  sint32 yaw_decidegree = 8;         // unit: 0.1 degree
  DjiUTC utc_time = 9;
  DeviceIdentify device_identify = 10;    // Include sn now.
}

// [end] Describe every data source as a message =============================

message DjiRealTimeTargetPoint {
  int64               timestamp = 1;
  uint32              frame_id = 2;
  DjiUTC              utc_time = 3;
  repeated string     source = 4;                     // The description of the one which create this point.
  message DjiTargetPointStruct {
    enum TARGET_POINT_TYPE {
      NO_USED = 0;
      TARGET_POINT = 1;                               // The target point create in local client.
      RNG_POINT = 2;                                  // The The laser ranging point is different from the target point. 
                                                      // This point is the center point of the screen detected by laser ranging in real time.*/
      ST_POINT = 3;                                   // Spotlight tracking point, the target position calculated in the camera automatic tracking mode,
                                                      // on the screen display, mutually exclusive with Rng, if there is this point, 
                                                      // the Rng point may not be displayed.
    }
    TARGET_POINT_TYPE type = 1;                       // The point type.
    uint32            index = 2;                      // The point index.
    double            latitude = 3;                   // unit: rad.
    double            longitude = 4;                  // unit: rad.
    float             altitude_ellipsoid = 5;         // unit: m. In ellipsoidal coordinate system.
    float             altitude_relative = 6;          // unit: m. The altitude relatived to take-off point.
    float             video_stream_window_x = 7;      // 0~1, Relative to the current position of the current stream viewport x,
                                                      // the upper left corner of the screen is 0, and negative numbers are used to indicate illegal values
                                                      // and are not displayed on the viewport.
    float             video_stream_window_y = 8;      // 0~1, Relative to the current position of the current stream viewport x,
                                                      // the upper left corner of the screen is 0, and negative numbers are used to indicate illegal values
                                                      // and are not displayed on the viewport.
    bool              altitude_ellipsoid_can_use = 9;
  }
  repeated DjiTargetPointStruct TargetPointStruct = 5;
}

message DjiFlyingWindState {
  int64  timestamp = 1;
  uint32 frame_id = 2;
  DjiUTC utc_time = 3;
  float  wind_speed = 4;          // unit:m/s, precision:one decimal place
  DIRECTION_TYPE wind_dir = 5;    // The direction of the wind
}

message DjiFlyingHomePoint {
  DjiGpsBasic point_pos_info = 3;
}

message DjiFlyingInherentInfo {
  DjiInherentInfoBase base_info = 1;
}

message DjiFlyingFlyOrderID {
  string              order_id = 1; // limited to 48 characters
}

message DjiRCInherentInfo {
  DjiInherentInfoBase base_info = 1;
  enum REMOTE_CONTROL_ROLES {
    RC_ROLES_A = 0;
    RC_ROLES_B = 1;
  }
  REMOTE_CONTROL_ROLES rc_roles = 2;     // [only remote control] The remote control position index
}

message DjiPayloadInherentInfo {
  DIRECTION_TYPE base_info = 1;
  enum PAYLOAD_PHY_POS {
    INDEX_1 = 0;
    INDEX_2 = 1;
    INDEX_3 = 2;
  }
  PAYLOAD_PHY_POS payload_phy_pos = 2;   // [only payload] The payload devices physical position index
}

dji_video_metadata.proto

// DJI Media Metadata using Protobuf 3

// version = 1

syntax = "proto3";
import "dvtm_library.proto";

message DjiVideoMetadata {
  DjiVideoGlobalInfo              video_global_info = 1;
  repeated DjiCameraBasic         camera_basic = 5;
  repeated DjiGpsBasic            gps_basic = 6;                // The old flying pos in camera, not suggest to use any more.
  repeated DjiFlyingState         flying_state = 7;             // Push frequency: 10Hz, Include flying speed and flying attitude.
  repeated DjiGimbal              gimbal = 8;                   // Push frequency: Hz.
  repeated DjiLaserRanging        laser_ranging = 9;
  DjiPayloadInherentInfo          payload_inherent_info = 10;   // Push frequency: 1Hz.

  repeated DjiFlyingWindState     flying_wind_state = 50;       // Push frequency: 10Hz.
  repeated DjiGpsBasic            flying_pos = 51;              // Push frequency: 10Hz, The position information of the flying, using GPS/RTK, etc.
  repeated DjiFlyingHomePoint     flying_home_point = 52;       // Push frequency: 10Hz.
  DjiRealTimeTargetPoint          dji_rt_target_point = 53;     // Push frequency: 2Hz.
  DjiFlyingInherentInfo           flying_inherent_info = 54;    // Push frequency: 1Hz.
  DjiFlyingFlyOrderID             flying_order_id = 55;         // Event trigger.

  repeated DjiGpsBasic            flying_rc_pos = 100;           // Push frequency: Hz. The position information of the remote control which control this flying.
  DjiRCInherentInfo               flying_rc_inherent_info = 101; // Push frequency: 1Hz.
}

Metadata 封装成一个单独 NALU，附加在每个 H264 视频 I 帧或 P 帧的末尾，NAL 类型为
0x06,其中 UUID 固定为

{0x81, 0x6d, 0x38, 0x4e, 0x99, 0x8c, 0x11, 0xea, 0xb2, 0x94, 0x02, 0xfc, 0xdc, 0x4e, 0x74, 0x12}

3. Metadata 解析指导

1. 将 metadata 数据，通过 protobuf 工具，序列化成二进制数据。
2. 根据 Metadta in H264 介绍的格式，将 metadata protobuf 二进制数据封装到H264/H265 的 SEI 帧里，紧随着每一个 I/P 帧传输。
3. 解析端收取的时候也是先收取到码流，根据码流解析出一个个帧，然后识别到 SEI帧，按照格式解析得到 metadata protobuf 二进制数据。最后根据 protobuf 格式(protobuf 支持多种语言多种平台)，解析出 metadata 数据结构体。
4. 解析这个环节最关键的三个要素：metadata protobuf file，protobuf 对应平台的编译工具，H264/H265 SEI 帧。

---

四、DJI GB28181 协议信令详解

抽空再写

总结

本文大部分资源来自于大疆内部文档，本人仅对其中的文件缺失进行了补全，最终解释权归大疆所有

双重加密的WeChat号，很简单的啦
MzkyODdmMTlhZWQzYzNmMTZjMWUxN2E0MDlmOWI0MDY