Any2Policy: Learning Visuomotor Policy with Any-Modality（类似AnyGPT）

发表时间：NeurIPS 2024

论文链接：https://readpaper.com/pdf-annotate/note?pdfId=2598959255168534016&noteId=2598960522854466816

作者单位：Midea Group

Motivation：Current robotic learning methodologies often focus on single-modal task specification and observation, thereby limiting their ability to process rich multi-modal information.（从多模态的角度切入）

Any2Policy 框架旨在处理多模态输入，分别在指令和观察级别单独或串联容纳它们。

我们设计了嵌入式对齐模块，旨在同步不同模态之间的特征，以及指令和观察，确保不同输入类型的无缝和有效的集成。

解决方法：为了解决这一限制，我们提出了一个名为 Any-to-Policy Embodied Agents 的端到端通用多模态系统。该系统使机器人能够使用各种模式处理任务，无论是在文本图像、音频图像、文本点云等组合中。

实现方式：我们的创新方法包括训练一个通用模态网络，该网络适应各种输入，并与策略网络连接以进行有效控制。

In summary, our contributions are the follows:

• We introduce any-to-policy models that enable a unified embodied agent to process various combinations of modalities, effectively facilitating instruction and perception of the world.

• We present novel embodied alignment learning techniques designed to seamlessly align instructions and observations, enhancing both the effectiveness and efficiency of policy learning.

• We offer a multi-modal dataset tailored for robotics, encompassing 30 distinct tasks. This dataset covers a wide spectrum of modalities in both instruction and observation.

实验：我们组装了一个包含30个机器人任务的综合真实数据集。

a real-world setting using our own collected dataset。

Simulation Evaluation: Franka Kitchen [ 92] uses text-image and ManiSkill2.

结论：该框架有效地处理并响应机器人任务的多模态数据。整个框架与其多模态数据集相结合，代表了体现 AI 领域的重大进步。