【具身智能】AgiBotWorld数据集介绍——智元百万真实机器人数据集

相关链接

Github：https://github.com/OpenDriveLab/AgiBot-World
数据集下载链接：https://huggingface.co/datasets/agibot-world/AgiBotWorld-Alpha
数据集官网：https://agibot-world.com/

机器人

  采集数据用到的机器人型号为远征A2-D（官网介绍链接：https://www.zhiyuan-robot.com/products/A2_D）

数据集格式

数据目录

data
├── task_info
│   ├── task_327.json
│   ├── task_352.json
│   └── ...
├── observations
│   ├── 327 # This represents the task id.
│   │   ├── 648642 # This represents the episode id.
│   │   │   ├── depth # This is a folder containing depth information saved in PNG format.
│   │   │   ├── videos # This is a folder containing videos from all camera perspectives.
│   │   ├── 648649
│   │   │   └── ...
│   │   └── ...
│   ├── 352
│   │   ├── 648544
│   │   │   ├── depth
│   │   │   ├── videos
│   │   ├── 648564
│   │   │   └── ...
│   └── ...
├── parameters
│   ├── 327
│   │   ├── 648642
│   │   │   ├── camera
│   │   ├── 648649
│   │   │   └── camera
│   │   └── ...
│   └── 352
│       ├── 648544
│       │   ├── camera # This contains all the cameras' intrinsic and extrinsic parameters.
│       └── 648564
│       │    └── camera
|       └── ...
├── proprio_stats
│   ├── 327[task_id]
│   │   ├── 648642[episode_id]
│   │   │   ├── proprio_stats.h5 # This file contains all the robot's proprioceptive information.
│   │   ├── 648649
│   │   │   └── proprio_stats.h5
│   │   └── ...
│   ├── 352[task_id]
│   │   ├── 648544[episode_id]
│   │   │   ├── proprio_stats.h5
│   │   └── 648564
│   │    └── proprio_stats.h5
│   └── ...

任务信息-json文件

  每个json文件存储了当前任务的描述信息，包括任务名称、初始关于场景的文本描述、任务对应的动作信息（action config）、关键帧（key frame）。
  其中动作信息包括：动作开始时对应的帧数、动作结束时对应的帧数、动作对应的文本描述、动作对应的技能；关键帧包括：关键帧开始时对应的帧数、关键帧结束时对应的帧数、关键帧对应的文本描述。

任务信息案例

[ {"episode_id": 649078,
   "task_id": 327,
   "task_name": "Picking items in Supermarket",
   "init_scene_text": "The robot is in front of the fruit shelf in the supermarket.",
   "lable_info":{
    "action_config":[
       {"start_frame": 0,
        "end_frame": 435,
        "action_text": "Pick up onion from the shelf."
        "skill": "Pick"
       },
       {"start_frame": 435,
        "end_frame": 619,
        "action_text": "Place onion into the plastic bag in the shopping cart."
        "skill": "Place"
       },
    ]
    "key_frame": [
      {"start": 0,
       "end": 435,
       "comment": "Failure recovery"
      }
    ]
},
]

注：动作描述（action text）和关键帧（key frame）并没有全部公开，计划一月中旬公开这些数据。

观测数据

  共包括两种数据，深度图数据和RGB图像数据。

深度图

  分辨率为640×480，数值范围为0-240。机器人介绍里说是有三个RGBD相机，但是目前公开的数据集中，深度图似乎只有头部中间相机的图像，两个手的深度图信息似乎没有给，不知道后面会不会给出。

RGB图像数据

  整个机器人共有8个相机，头部中间的相机、头部中间的鱼眼相机、头部左侧的鱼眼相机、头部右侧的鱼眼相机、后脑勺左侧鱼眼相机、后脑勺右侧鱼眼相机、左手相机、右手相机（共包括3个普通相机，5个鱼眼相机），分辨率均为960×768。

头部中间的相机

头部中间的鱼眼相机

后脑勺左侧鱼眼相机

右手相机

机器人本体状态数据

|-- timestamp
|-- state
    |-- effector
        |-- force
        |-- position
    |-- end
        |-- angular
        |-- orientation
        |-- position
        |-- velocity
        |-- wrench
    |-- head
        |-- effort
        |-- position
        |-- velocity
    |-- joint
        |-- current_value
        |-- effort
        |-- position
        |-- velocity
    |-- robot
        |-- orientation
        |-- orientation_drift
        |-- position
        |-- position_drift
    |-- waist
        |-- effort
        |-- position
        |-- velocity
|-- action
    |-- effector
        |-- force
        |-- index
        |-- position
    |-- end
        |-- orientation
        |-- position
    |-- head
        |-- effort
        |-- position
        |-- velocity
    |-- joint
        |-- effort
        |-- index
        |-- position
        |-- velocity
    |-- robot
        |-- index
        |-- orientation
        |-- position
        |-- velocity
    |-- waist
        |-- effort
        |-- position
        |-- velocity

术语解释

状态和动作

• 状态（state）：不同传感器（sensor）和执行器（actuator）的监测信息；
• 动作（action）：发送到硬件层的指令信息，控制器将对这些指令做出响应；发出的指令和实际执行的状态之间存在差异

执行器

• 效应器（effector）：指末端效应器，例如灵巧的手或抓手；
• 末端（end）：机器人手指上的6自由度（6DoF）末端位姿
• 头部（head）：为机器人的头部视角，有两个自由度（俯仰角pitch和偏航角yaw，也就是上下角度和左右角度）
• 关节（joint）：指机器人双臂的关节，共有14个自由度，每只手臂有7个自由度；
• 机器人（robot）：指机器人在其周围环境中的姿态，方向和位置是指机器人在测程坐标系（odometry coordinate）中的相对姿态，其中原点是在机器人通电之后设置的；
• 腰部（waist）：指机器人的腰部关节，有两个自由度（俯仰pitch和升降lift）

公共属性

• 位置（position）：包括空间位置、编码器位置、角度等等；
• 速度（velocity）：指速度（speed）；
• 角速度（angular）：指角速度（angular velocity）；
• 力（effort）：指电机的扭矩（Torque of the motor），（目前暂时没有）；
• 扳手（wrench）：6维力，xyz方向上的力和扭矩（force in the xyz directions, and torque），（目前暂时没有）；

每个数据的维度和意义

Group	Shape	Meaning
/timestamp	[N]	timestamp in nanoseconds
/state/effector/position (gripper)	[N, 2]	left [:, 0], right [:, 1], gripper open range in mm
/state/effector/position (dexhand)	[N, 12]	left [:, :6], right [:, 6:], joint angle in rad
/state/end/orientation	[N, 2, 4]	left [:, 0, :], right [:, 1, :], flange quaternion with xyzw
/state/end/position	[N, 2, 3]	left [:, 0, :], right [:, 1, :], flange xyz in meters
/state/head/position	[N, 2]	yaw [:, 0], pitch [:, 1], rad
/state/joint/current_value	[N, 14]	left arm [:, :7], right arm [:, 7:]
/state/joint/position	[N, 14]	left arm [:, :7], right arm [:, 7:], rad
/state/robot/orientation	[N, 4]	quaternion in xyzw, yaw only
/state/robot/position	[N, 3]	xyz position, where z is always 0 in meters
/state/waist/position	[N, 2]	pitch [:, 0] in rad, lift [:, 1]in meters
/action/*/index	[M]	actions indexes refer to when the control source is actually sending signals
/action/effector/position (gripper)	[N, 2]	left [:, 0], right [:, 1], 0 for full open and 1 for full close
/action/effector/position (dexhand)	[N, 12]	same as /state/effector/position
/action/effector/index	[M_1]	index when the control source for end effector is sending control signals
/action/end/orientation	[N, 2, 4]	same as /state/end/orientation
/action/end/position	[N, 2, 3]	same as /state/end/position
/action/end/index	[M_2]	same as other indexes
/action/head/position	[N, 2]	same as /state/head/position
/action/head/index	[M_3]	same as other indexes
/action/joint/position	[N, 14]	same as /state/joint/position
/action/joint/index	[M_4]	same as other indexes
/action/robot/velocity	[N, 2]	vel along x axis [:, 0], yaw rate [:, 1]
/action/robot/index	[M_5]	same as other indexes
/action/waist/position	[N, 2]	same as /state/waist/position
/action/waist/index	[M_6]	same as other indexes

数据用法

  对于这种用设备，一步一步采集的数据，在执行动作的每一帧，我们都可以获得传感器上的各个响应数据，例如位置、朝向、关节夹角等，我们将这些传感器数据统称为状态数据（state），同时，我们也可以捕获每个关节的动作信息（action），包括位置、速度等等。模型所要做的就是根据当前的图像观测observation、当前的状态state，去预测下一刻的动作action。
  智元官方给出了将公开数据的格式转化为LeRobot数据格式的代码（LeRobot链接：https://github.com/huggingface/lerobot），LeRobot是一个比较主流的机器人数据存储格式（类似目标检测中的COCO格式），将各种状态、动作信息统一到一个文件夹里，便于后续的代码调用。

  LeRobot数据格式的转换代码：https://huggingface.co/datasets/agibot-world/AgiBotWorld-Alpha/blob/main/scripts/convert_to_lerobot.py

  以任务327为例，该任务相关的状态维度为20，动作空间的维度为22：

所有任务

  一共45个任务，具体如下：
task_name: Supermarket pickup - Produce area.
init_scene_text: A robot is in front of the fruit shelf in the supermarket.
task_name: Opening fridge and taking out food.
init_scene_text: A robot is placed in front of a refrigerator with the door closed.
task_name: Supermarket pickup - Shelf area.
init_scene_text: A robot is in front of the supermarket shelf, with a shopping cart next to it.
task_name: Hanging clothes with a hanger.
init_scene_text: A robot stands beside the bed or sofa, with clothes and a hanger placed on the bed or sofa.
task_name: Supermarket packing - Hard.
init_scene_text: A robot is in front of the checkout counter, with three different types of items and a shopping bag on the counter.
task_name: Washing dishes with a dishwasher.
init_scene_text: The dishwasher lid is slightly closed, with items to be cleaned placed in the right sink.
task_name: Toasting bread.
init_scene_text: A bowl containing two slices of toast or a pack of bread slices is placed in front of the robot on the left side, while a toaster is placed in front of the robot on the right side.
task_name: Sorting household items.
init_scene_text: A robot stands in front of a table, with a 2-compartment material box and a target box on the table. The target box is on the right, and the 2-compartment material box is on the left.
task_name: Supermarket packup - Easy.
init_scene_text: A robot is in front of the checkout counter, with three different types of items and a shopping bag on the counter.
task_name: Flattening shorts.
init_scene_text: A robot is placed in front of a bed in the bedroom, with a pair of shorts messily placed on the bed.
task_name: Folding shorts.
init_scene_text: A pair of shorts is laid flat horizontally on a bed in the bedroom.
task_name: Sorting laundry and personal care products - task2.
init_scene_text: A robot stands in front of a table, with a 3-compartment material box and a target box on the table. The target box is on the left, and the 3-compartment material box is on the right.
task_name: Food sorting.
init_scene_text: A robot stands in front of a table, with a 4-compartment material box and a target box on the table. The target box is on the left, and the 4-compartment material box is on the right.
task_name: Taking out toast.
init_scene_text: A robot has an empty dinner plate on the left and a toaster with two pieces of toasted bread on the right.
task_name: Barbecuing.
init_scene_text: A robot stands in front of an oven with a baking tray containing two pieces of vegetables to be grilled. The oven is placed on the left side, and a oil dish and seasoning bottles are placed on the right side.
task_name: Magnet block packing.
init_scene_text: The tabletop is arranged with a nine-compartment material tray, and to the right of the tray on the assembly line, there are 1 to 5 magnetic blocks placed.
task_name: Supermarket packing - Medium.
init_scene_text: A robot is in front of the checkout counter, with four different types of items and a soft woven bag on the counter.
task_name: Sorting laundry and personal care products - task1.
init_scene_text: A robot stands in front of a table, with a 3-compartment material box and a target box on the table. The target box is on the left, and the 3-compartment material box is on the right.
task_name: Making tea.
init_scene_text: A box of tea bags, a cup, and a kettle are placed on the table.
task_name: Sorting electronic products.
init_scene_text: A robot stands in front of a table, with a 4-compartment material box and a target box on the table. The material box is on the left, and the target box is on the right.
task_name: E-commerce packing - Task 1.
init_scene_text: A robot stands in front of a table, with a cardboard box, air column packaging, and items placed on the table in order.
task_name: Cleaning up the dining table.
init_scene_text: The robot holds an empty dinner plate with a fork in a bowl, and a plate, cup, and napkin are placed on the table.
task_name: E-commerce packing - Task 2.
init_scene_text: A robot stands in front of a table, with a cardboard box, air column packaging, and items placed on the table in order.
task_name: Inserting a book into a bookshelf.
init_scene_text: A robot stands in front of a bookshelf, and there is a slight gap between each book on one shelf. The books to be inserted are placed flat on another shelf.
task_name: Supermarket pickup - Freezer area.
init_scene_text: A robot is in front of the freezer section of the supermarket.
task_name: Supermarket pickup - Fresh area.
init_scene_text: A robot is in front of the fresh food section of the supermarket, with a shopping cart next to it.
task_name: Scanning items at a supermarket checkout.
init_scene_text: A robot stands at the supermarket checkout counter, with three items and a barcode scanner on the counter. The items are randomly placed.
task_name: Washing bottles.
init_scene_text: A robot faces the sink, with a bottle of colored liquid on the countertop and a cup brush standing in a storage box on the countertop.
task_name: Sorting clothes.
init_scene_text: A robot stands in front of a table, with a material box containing clothes and a target box on the table. The 2-compartment material box is on the right, and the target box is on the left.
task_name: Pouring water in a restaurant.
init_scene_text: A robot’s operational area includes an empty cup and a bottle of colored tea in the kettle.
task_name: Opening the wardrobe to hang clothes.
init_scene_text: The robot is facing the wardrobe, with its right hand holding the clothes to be hung.
task_name: Industrial logistics packup (picking items from a conveyor belt).
init_scene_text: A conveyor belt, a logistics box, and air column packaging are placed on the table, with the logistics box on the left and the air column packaging on the right.
task_name: Cleaning countertop trash.
init_scene_text: There are three pieces of trash on the countertop, with a trash can next to it. The trash can has a trash bag inside.
task_name: Inserting a memory kit.
init_scene_text: A robot stands in front of the table, holding a memory kit in its left hand.
task_name: Using the washing machine to wash clothes.
init_scene_text: There is a front-load washing machine in front of the robot, and a laundry basket next to it. Two pieces of clothing are draped over the edge of the laundry basket.
task_name: Putting the pen into the pen holder.
init_scene_text: The robot is facing a desk, with pen 1, pen 2, and a pen holder randomly placed on the tabletop.
task_name: Ironing clothes.
init_scene_text: A robot stands in front of a garment steamer with a piece of clothing hanging on it, while the steam iron is on its stand.
task_name: Hanging clothes with a hanger.
init_scene_text: The robot stands by the bed or sofa, with clothes and a hanger placed on the bed or sofa. Two neatly folded clothes are placed around, or nothing is placed around.
task_name: Dual-robot table carrying.
init_scene_text: Two robots are on opposite sides of the table, facing each other, with the table in between.
task_name: Making a sandwich.
init_scene_text: The robot is facing a cooking counter. To the robot’s left-front, there is a toast basket containing slices of toast. To its right, there is a plate with vegetable leaves and meat slices, and a bottle of salad dressing. In front of the robot, there is an empty plate and a toaster.
task_name: Ironing clothes.
init_scene_text: An ironing machine is leaning against the wall. The robot is standing in front of the ironing machine. A short-sleeved shirt is hanging on the ironing machine, and a iron rests on the stand of the ironing machine.