基于大模型的自行车道CAD设计系统

LLM（大型语言模型）是强大的工具。对于许多人来说，用语言表达愿望通常比浏览复杂的 GUI 更简单。我以“构建 LLM 支持的应用程序”课程为契机，用人类语言提示取代了传统的车道CAD构建系统。

NSDT工具推荐： Three.js AI纹理开发包 - YOLO合成数据生成器 - GLTF/GLB在线编辑 - 3D模型格式在线转换 - 可编程3D场景编辑器 - REVIT导出3D模型插件 - 3D模型语义搜索引擎 - Three.js虚拟轴心开发包 - 3D模型在线减面 - STL模型在线切割 

1、系统简介和环境搭建

我受雇于 urb-x.ch，这是一家专门从事自行车道建设的公司。我们的轨道采用模块化构建块进行独特设计，可以通过多种方式进行组合。当拼凑在一起时，整个轨道网络形成了这些构建块的树形结构。

在某些方面，这个任务可以看作是一个视频游戏沙盒。可以将其视为在《过山车大亨》或《赛道狂热》中建造轨道。然而，虽然这些游戏提供了直观的设计体验，但它们无法提供土木工程项目所需的精度水平。另一方面，传统的 CAD 程序不允许大规模使用这样的模块化系统。

我们已经建立了一个无/低代码系统，让你可以轻松地创建车道，该系统构建在 Rhino 和 Grasshopper 的基础上。收获是什么？你必须投入一些时间来熟悉可用的组件。

与大多数视频游戏一样，与系统的交互可以被视为数据库系统中的事务。为此，我设计了一个简单的 json/ pydantic 模式来存储轨道的配置。现在更新变成了简单的配置更新。我们将此配置称为“TrackConfig”。

{
  "version": "0.0.2",
  "library_dir": [
    "/Users/lucas/projects/gh/"
  ],
  "origin": {
    "position": [
      0.0,
      0.0,
      0.0
    ],
    "rotation": [
      0.0,
      0.0,
      0.0
    ]
  },
  "global_upgrades": {
    "PV": true,
    "HEATING": true,
    "FE3": {
      "LR": false
    }
  },
  "instance": {
    "id": "Oliver",
    "part_name": "XC50-4L",
    "children": {
      "-1": {
        "id": "Liam",
        "part_name": "XC50-4R",
        "children": {
          "-1": {
            "id": "Oliver",
            "part_name": "XC50-4R",
            "children": {
              "-1": {
                "id": "Ethan",
                "part_name": "XC50-4R",
                "children": {}
              }
            }
          }
        }
      }
    }
  },
  "configs": {},
  "selected": "Oliver"
}

作为一个项目，我现在使用新的 OpenAI 函数来解析模型输出/实际上限制模型输出以匹配预期格式。

最初，我的目标是直接生成新的 TrackConfig。这种方法效果很好，但有一个显着的缺点：延迟非常大。不过，这还不错：大约 15 秒，这在大多数情况下仍然比使用 GUI 更快。总而言之，我发现这对于迭代过程来说是不可接受的，并且希望加快速度。我使用 W&B Prompts 记录所有内容，显示延迟和许多其他指标。它们工作得非常好，特别是在深入研究模型结果时！

为了应对延迟挑战，我的策略是减少模型所需的输出。这种方法的结果是集成了简单的更新功能。

from typing import Any, Optional, Union


from PythonModules.schema import TrackConfig, Part, PartNames


from pydantic import BaseModel, Field


class UpdatePart(BaseModel):
    """Updating or creating a part with the given id and part_name"""


    reasoning: str = Field(..., description="Answer all reasoning questions")
    id: str = Field(..., description="The unique id of a part. This is a unique random english first name.")
    part_name: PartNames = Field(..., description="The name of the part e.g. 'XS12'")
    parent: str = Field("", description="The id of the parent part or '' if the part is the root part. Usually This is the current selected part.")
    index: int = Field(-1, description="Where to insert the new part. -1 means append.")


    def __call__(self, track_config: Optional[TrackConfig] = None):
        return #code ommited




class UpdateSelection(BaseModel):
    """Select the part with the given id"""


    reasoning: str = Field(..., description="Which id's parts could be selected? Why does each part make sense or not?")
    id: str = Field(..., description="The unique id of a part. This is a unique random english first name.")


    def __call__(self, track_config: Optional[TrackConfig] = None):
        print(f"UpdateSelection(reasoning={self.reasoning}, id={id})")
        return #code ommited


# Other possible updates

虽然LLM很强大，但它们有一个一致的缺点：零样本预测可能不可靠。我知道带有推理的输出要可靠得多。不幸的是，目前还没有任何 LangChain 链能够提供推理与 OpenAI 函数调用的组合。

我公认的黑客解决方案（从头开始，工程解决方案😉）是在函数调用中强制使用“推理”字段。前面讨论的 pydantic 模式说明了一个典型的例子。

这种策略虽然非常规，但已被证明是无价的，并且可能成为类似应用程序的关键要点。

现在，讨论延迟。如果没有推理组件，延迟会徘徊在 1.5 秒左右，这个值低得惊人。添加推理后，延迟会根据我所采用的推理的深度和数量而变化。本质上，增加的运行时间与推理标记的数量成线性比例。那里相当简单。大多数情况下，我的延迟时间都小于 4 秒，尽管优化仍在进行中并且我希望能将其降低。

借助 W&B Prompts，我可以制作提示并观察其他用户如何与我的工具交互。这种洞察力使我能够分析两种提示的功效，从而进行改进，例如改进系统提示以增强预测。

让我们深入探讨提示业务。由于 GPT-3 Turbo 的上下文大小相当大，而我的预测令牌数量相当少，我有一些空间可以使用，所以我可以在提示上挥霍。

我把所有我认为重要的东西都扔掉了，保持干净整洁。查看下面我进行的设置：

system_template = """
You are a world class assistant, helping an engineer to build an amazing street network from prefefined parts. The whole system is modular. 
You are responsible for deciding on which updates to do to the underlying configuration. The engineer will then update the configuration based on your input. 
You get access to a pointer named 'selected'. This id shows you relative to what you probably should append things to.
The whole track configuration is stored in a json file. In the instance key a tree of parts exists. The decendant of a child is the "children" field. The main child has key "-1".
The whole track consisting of a tree should make it easier to describe on where to append things.


The configuration is based on the following pydantic schema:


class PartNames(str, Enum):
    Straigt30mElement = "XS12"
    CurveRight50mRadius10mLenthgElement = "XC50-4L"
    CurveLeft50mRadius10mLenthgElement = "XC50-4R"
    DeletedElement = "Deleted"


class Origin(BaseModel):
    position: List[float] = Field(..., description="The origin position coordinates (x,y,z))")
    rotation: List[float] = Field(..., description="The origin rotation angles in degrees (x,y,z)")
    
class Part(BaseModel):
    id: str = Field(..., description="The unique id of a part. This is a unique random english first name.")
    part_name: PartNames = Field(..., description="The name of the part")
    children: Optional[Dict[str, 'Part']] = Field({}, description="Dictionary of child parts. Key=='next' is the default next part")


Part.update_forward_refs()  # This updates Element to include the recursive reference to itself


class TrackConfig(BaseModel):
    version: str = Field(..., description="The version of the track configuration")
    library_dir: List[str] = Field(..., description="List of paths to the libraries")
    origin: Origin = Field(..., description="The origin position and rotation of the track")
    global_upgrades: Dict[str, Any] = Field(..., description="Global upgrades for every element")
    instance: Part = Field(..., description="Part instances of the track")
    configs: Dict[str, Dict[str, Any]] = Field(..., description="Configuration dictionary for the track configuration")
    selected: str = Field(..., description="The id of the selected Element")


For new elements think of a new english first name as id! Think of a random name.
If not specified otherwise append or add new parts to the selected part. Never add new parts to a parent as a different child then -1, unless specified otherwise!
If a parent has other children it usually is asumed that the child with key -1 is the main child and that you should append to that one or one of its children.


Altough also visible in the schema, the following part_name's are available:
Straigt 30m Element: "XS12"
Curve Right!! 50m Radius 10m outer Length Element: "XC50-4L"
Curve Left!!  50m Radius 10m outer Length Element: "XC50-4R"


Reasons for the update fields should answer the following questions! You have to answer all of them:
- Which id do you want to update or is it a new id?
- What part_name do you want to use?
- Why this part_name?
- What parents could you use?
- If the parent already has children, why do you want to replace main child?


"""


prompt_template = """
The current configuration is as follows:
--------------------
{track_config}
--------------------
Use the given format to pick update steps from the following input:
--------------------
{input}
--------------------
Tips: Answer all questions.
Tips: Make sure to answer in the correct format!!!
"""

现在让我们看看模型调用：

class Track:
    def __init__(self, llm, system_template, prompt_template):
        self.path = "track.json"
        self.new_config = None
        prompt_msgs = [
            SystemMessage(content= system_template),
            HumanMessagePromptTemplate.from_template(prompt_template)
        ]
        prompt = ChatPromptTemplate(messages=prompt_msgs)


        self.chain = create_openai_fn_chain([UpdatePart,UpdateSelection], llm, prompt, verbose=False)


    @property
    def track_config(self):
        if not self.new_config:
            data_dict = json.load(open(self.path, "r"))
            track_config = TrackConfig(**data_dict)
            self.new_config = track_config


        return self.new_config
    
    @track_config.setter
    def track_config(self, track_config: TrackConfig):
        self.new_config = track_config
        
    def store(self):
        json.dump(self.track_config.dict(), open(self.path, "w"), indent=2)
    
    def agent(self):
        # interactively update the track_config
        # asks for user input and calls the callback function


        while prompt:=input("How should the track be updated? to quit press enter without input."):
            self(prompt)
    
    # Possibly wrap the call in retries if the to predict format is very difficult
    # @retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
    def __call__(self, input) -> Any:
        json_str = json.dumps(self.track_config.dict(), indent=2)
        function:Union[UpdatePart,UpdateSelection] = self.chain.run(input = input, track_config=json_str)
        
        # call function with the track_config
        res = function(self.track_config)


        # store the new track_config
        self.store()
        
        return res


model = "gpt-3.5-turbo"
llm = ChatOpenAI(model=model,temperature=0.9)
track = Track(llm, system_template, prompt_template)

现在我们要么使用 track.agent() 方法来连续更新配置。或者使用单个命令：

track("append a new right element to the track")
track("Add a straight part to the last added track")
...

2、我如何使用 W&B 提示

此处使用 W&B Prompts 主要是为了记录并查看其他人如何使用我的工具。这是一次了解人们在建造轨道时所做出的不同假设的旅程，并了解模型提出的古怪假设。您可以在函数调用的“原因”字段中发现这些模型假设。

伙计，我完全赞成这一切是多么的轻松。只需设置一个环境变量，然后就可以开始了。

# we need a single line of code to start tracing LangChain with W&B
os.environ["LANGCHAIN_WANDB_TRACING"] = "true"

好吧，说实话：现在，我的锁链只是一场单步舞。如果我能把它分成几个阶段，特别是分离出推理，那就很巧妙了。但是，嘿，这是一项正在进行的工作。我很想看看其他人在铺设轨道时如何使用我的工具。

以下是 CAD 中车道的形状：

注意：我们只是在这里展示车道。为了清晰起见，所有额外的东西，如房屋、街道和桥梁，都没有显示在图中。

3、未来的工作

是时候让事情活跃起来了！我的目标是为动态放置的东西添加配置更新 - 想想灯、柱子和一些华丽的升级，比如环形交叉口。好消息？模式已经准备好了。

好吧，事情是这样的：当模型面对深度嵌套的结构时，它会有点绊倒。可能需要砍掉树并在这些树节点之间变得更加线性。

而且，最重要的是，我想让那些非技术人员轻松完成设置；）

---

原文链接：基于LLM的自行车道CAD - BimAnt