【LLM+三维场景】3D-GPT: Procedural 3D MODELING WITH LARGE LANGUAGE MODELS

3D-GPT: Procedural 3D MODELING WITH LARGE LANGUAGE MODELS 3D-GPT： 使用大型语言模型进行程序化 3D 建模

2023.10

论文主页
论文地址
代码地址（未放）
一句话生成3D世界，未公布代码已获141星！或引发3D建模行业革命

Abstract

In the pursuit of efficient automated content creation, procedural generation, leveraging modifiable parameters and rule-based systems, emerges as a promising approach. Nonetheless, it could be a demanding endeavor, given its intricate nature necessitating a deep understanding of rules, algorithms, and parameters. To reduce workload, we introduce 3D-GPT, a framework utilizing large language models~(LLMs) for instruction-driven 3D modeling. 3D-GPT positions LLMs as proficient problem solvers, dissecting the procedural 3D modeling tasks into accessible segments and appointing the apt agent for each task. 3D-GPT integrates three core agents: the task dispatch agent, the conceptualization agent, and the modeling agent. They collaboratively achieve two objectives. First, it enhances concise initial scene descriptions, evolving them into detailed forms while dynamically adapting the text based on subsequent instructions. Second, it integrates procedural generation, extracting parameter values from enriched text to effortlessly interface with 3D software for asset creation. Our empirical investigations confirm that 3D-GPT not only interprets and executes instructions, delivering reliable results but also collaborates effectively with human designers. Furthermore, it seamlessly integrates with Blender, unlocking expanded manipulation possibilities. Our work highlights the potential of LLMs in 3D modeling, offering a basic framework for future advancements in scene generation and animation.

在追求高效的自动内容创建过程中，利用可修改参数和基于规则的系统进行程序生成是一种很有前途的方法。

然而，由于其复杂性，需要对规则、算法和参数有深入的了解，这可能是一项艰巨的工作。

为了减少工作量，我们引入了 3D-GPT 框架，该框架利用大型语言模型（LLM）进行指令驱动的 3D 建模。

3D-GPT 将大型语言模型定位为熟练的问题解决者，将程序化三维建模任务分解为可访问的片段，并为每个任务指定合适的agent。

3D-GPT 集成了三个核心agent：

• 任务派遣agent;
• 概念化agent;
• 建模agent。

它们共同实现了两个目标：

• 首先，它增强了简洁的初始场景描述，将其发展为详细的形式，同时根据后续指令动态调整文本。
• 其次，它整合了程序生成功能，从丰富的文本中提取参数值，从而轻松地与三维软件对接，进行资产创建。

我们的实证调查证实，3D-GPT 不仅能解释和执行指令，提供可靠的结果，还能与人类设计师有效协作。此外，它还能与 Blender 无缝集成，从而实现更多的操作可能性。我们的工作彰显了 LLM 在三维建模中的潜力，为未来场景生成和动画制作的进步提供了一个基本框架。

简评

概念很有意思，但是实际很简单，本质为blender + python，由gpt生成python代码。加上项目未发布开源代码，噱头略大于实际。