【Prompts + 图像生成视频】Follow-Your-Click: Open-domain Regional Image Animation via Short Prompts

Follow-Your-Click: Open-domain Regional Image Animation via Short Prompts 跟随您的点击 通过简短提示制作开放域区域图像动画

2024.3.13

论文地址
代码地址
项目地址

区域动画+短提示：Follow-Your-Click引领图像动画新潮流

Abstract

Despite recent advances in image-to-video generation, better controllability and local animation are less explored. Most existing image-to-video methods are not locally aware and tend to move the entire scene. However, human artists may need to control the movement of different objects or regions. Additionally, current I2V methods require users not only to describe the target motion but also to provide redundant detailed descriptions of frame contents. These two issues hinder the practical utilization of current I2V tools. In this paper, we propose a practical framework, named Follow-Your-Click, to achieve image animation with a simple user click (for specifying what to move) and a short motion prompt (for specifying how to move). Technically, we propose the first-frame masking strategy, which significantly improves the video generation quality, and a motion-augmented module equipped with a short motion prompt dataset to improve the short prompt following abilities of our model. To further control the motion speed, we propose flow-based motion magnitude control to control the speed of target movement more precisely. Our framework has simpler yet precise user control and better generation performance than previous methods. Extensive experiments compared with 7 baselines, including both commercial tools and research methods on 8 metrics, suggest the superiority of our approach. Project Page: https://follow-your-click.github.io/

尽管图像到视频生成技术近年来不断进步，但对更好的可控性和局部动画的探索却较少。

这两个问题阻碍了当前 I2V 工具的实际应用：

• 大多数现有的图像到视频生成方法都不具备局部感知能力，往往会移动整个场景。然而，人类艺术家可能需要控制不同物体或区域的移动。

• 此外，目前的 I2V 方法不仅需要用户描述目标运动，还需要提供冗余的帧内容详细描述。

在本文中，我们提出了一个名为 "Follow-Your-Click "的实用框架，通过用户简单的点击（指定要移动的内容）和简短的运动提示（指定如何移动）来实现图像动画。

• 在技术上，我们提出了第一帧屏蔽策略，该策略显著提高了视频生成质量；
• 我们还提出了运动增强模块，该模块配备了简短运动提示数据集，以提高模型的简短提示跟随能力。
  为了进一步控制运动速度，我们提出了基于流量的运动幅度控制，以更精确地控制目标的运动速度。

与之前的方法相比，我们的框架具有更简单而精确的用户控制和更好的生成性能。与包括商业工具和研究方法在内的 7 种基线方法在 8 个指标上进行的广泛实验比较表明，我们的方法更胜一筹。

项目页面： https://follow-your-click.github.io/

Results