【LLM+GUI智能】UFO: A UI-Focused Agent for Windows OS Interaction

UFO: A UI-Focused Agent for Windows OS Interaction

论文地址
代码地址
UFO: Windows操作系统交互的UI聚焦智体

Abstract

We introduce UFO, an innovative UI-Focused agent to fulfill user requests tailored to applications on Windows OS, harnessing the capabilities of GPT-Vision. UFO employs a dual-agent framework to meticulously observe and analyze the graphical user interface (GUI) and control information of Windows applications. This enables the agent to seamlessly navigate and operate within individual applications and across them to fulfill user requests, even when spanning multiple applications. The framework incorporates a control interaction module, facilitating action grounding without human intervention and enabling fully automated execution. Consequently, UFO transforms arduous and time-consuming processes into simple tasks achievable solely through natural language commands. We conducted testing of UFO across 9 popular Windows applications, encompassing a variety of scenarios reflective of users’ daily usage. The results, derived from both quantitative metrics and real-case studies, underscore the superior effectiveness of UFO in fulfilling user requests. To the best of our knowledge, UFO stands as the first UI agent specifically tailored for task completion within the Windows OS environment. The open-source code for UFO is available on https://github.com/microsoft/UFO.

我们介绍的 UFO 是一种创新的以用户界面为中心的智能体，它利用 GPT-Vision 的功能，为 Windows 操作系统上的应用程序量身定制，以满足用户的要求。

UFO 采用双智能体框架，对图形用户界面（GUI）和 Windows 应用程序的控制信息进行细致的观察和分析。这使智能体能够在单个应用程序内和跨应用程序之间无缝导航和操作，以满足用户的要求，即使在跨多个应用程序的情况下也是如此。该框架包含一个控制交互模块，便于在没有人工干预的情况下进行操作，并实现全自动执行。因此，UFO 可将艰巨耗时的流程转化为只需通过自然语言命令即可完成的简单任务。

我们在 9 个流行的 Windows 应用程序中对 UFO 进行了测试，其中包括反映用户日常使用情况的各种场景。通过量化指标和实际案例研究得出的结果表明，UFO 在满足用户请求方面具有卓越的功效。据我们所知，UFO 是首个专为 Windows 操作系统环境下完成任务而定制的用户界面代理。