Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields

Abstract

3D scene representations have gained immense popularity in recent years. Methods that use Neural Radiance fields are versatile for traditional tasks such as novel view synthesis. In recent times, some work has emerged that aims to extend the functionality of NeRF beyond view synthesis, for semantically aware tasks (such as editing and segmentation) using 3D feature field distillation from 2D foundation models. However, these methods have two major limitations: (a) they are limited by the rendering speed of NeRF pipelines, and (b) implicitly represented feature fields suffer from continuity artifacts reducing feature quality. Recently, 3D Gaussian Splatting has shown SOTA performance on real-time radiance field rendering.

In this work, we go one step further: in addition to radiance field rendering, we enable 3D Gaussian splatting on arbitrary-dimension semantic features via 2D foundation model distillation. This translation is not straightforward: naively incorporating feature fields in the 3DGS framework leads to warp-level divergence.

Our proposed method is general, and our experiments showcase novel view semantic segmentation, language-guided editing and segment anything through learning feature fields from SOTA 2D foundation models(such as SAM and CLIPLSeg).

提出的方法是通用的，实验展示了新颖的视图语义分割、语言指导的编辑和通过从SAM和CLIPLSeg等最先进的2D基础模型中学习特征字段来分割任何东西。

Additionally, to the best of our know