【读论文】Gaussian-based街景重建 HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting

1. HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting

1.1 What

Without LiDAR scans and 3D bounding boxes input, from images, noisy 2D semantic labels, and noisy optical flow, this paper involves the joint optimization of geometry, appearance, semantics, and motion using a combination of static and dynamic 3D Gaussians, where moving object poses are regularized via physical constraints.

1.2 Why

Introduction:

• With the rise of neural rendering, many approaches have emerged to lift 2D information to 3D space, but those works addressing dynamic scenes need additional bboxes.
• For PNF, naive joint optimization of per-frame pose transformations is prone to local minima and sensitive to initialization.
• Some works fit 2D situations but are non-trivial to extract accurate semantics in 3D due to the inaccurate (inferred) 3D geometry.

Related work:

• 3D Scene Understanding
  • Numerous techniques have focused on predicting semantic labels [5, 9, 35], depth maps [10, 28], and optical flows [42] solely from 2D input images. But they are limited in understanding 3D.
  • Another line of approach suggests conducting semantic scene understanding solely based on 3D input [29, 31]. But LiDAR is expensive.
  • NeRF-based works such as Semantic-NeRF are only applicable to static environments.
• Urban Scene Reconstruction
  • Many of these approaches rely on the availability of accurate 3D bounding boxes, as seen in NSG, MARS, and UniSim.
  • SUDS [36], avoids the use of 3D bounding boxes by grouping the scene based on learned feature fields.

1.3 How

Seeing from the pipeline, we know that the scene was divided into several dynamic and one static scenes. In static scene, three types of Gaussian were used to represent the scene and only two types of Gaussian were used in dynamic scenes. This is because the optical flow can be calculated from the motion of the Gaussian center. And then, just like NSG, the dynamic Gaussians are transformed to the whole scene, combining with the static Gaussians to obtain the rendered result of three types of Gaussian, which are supervised to learn.

1.3.1 Decomposed Scene Representation

1. Static and Dynamic 3D Gaussians

We know to divide the scene into static and dynamic, but the division basis still needs to be determined (Maybe refer to NSG).

Another thing is to add semantic logits $\mathbf{s}\in {\mathbb{R}}^S$ to each Gaussian, allowing for rendering 2D semantic labels.

2. Unicycle Model

The vehicle is represented by a unicycle model as below, with three elements: $(x_t,y_t,{\theta }_t)$.

Concretely,

$$\begin{array}{rl}& x_{t+1}=x_t+\Delta x=x_t+R(\sin {\theta }_{t+1}-\sin {\theta }_t)=x_t+\frac{v_t}{{\omega }_t}(\sin {\theta }_{t+1}-\sin {\theta }_t)\\ & y_{t+1}=y_t-\frac{v_t}{{\omega }_t}(\cos {\theta }_{t+1}-\cos {\theta }_t)\\ & {\theta }_{t+1}={\theta }_t+{\omega }_t\end{array}.$$
This model integrates physical constraints when compared to directly optimizing the transformations of dynamic vehicles at every frame independently.

1.3.2 Holistic Urban Gaussian Splatting

Three types of results—render images, semantic maps, and optical flow were rendered in this model.

1. Novel View Synthesis

The same as the original GS, but adds the effect of exposure.

That is because when working with 3D Gaussians, there is no neural network capable of processing appearance embeddings thereby compensating exposure. So inspired by Urban-NeRF, this paper generates an affine matrix $\mathbf{A}\in {\mathbb{R}}^{3\times 3}$ and vector $\mathbf{b}\in {\mathbb{R}}^3$ via a small MLP for each camera to add the exposure.

$$\tilde{\mathrm{C}}=\mathrm{A}\times \mathrm{C}+\mathrm{b}.$$

Then utilizing $\alpha$-blending:

$$\pi :\mathbf{C}=\sum _{i\in \mathcal{N}}{\mathbf{c}}_i{\alpha }_i^{\prime }\prod _{j=1}^{i-1}(1-{\alpha }_j^{\prime }).$$

• Explanation
  1. For the first sample (closest to the viewer), its color is weighted fully by its alpha value because there are no preceding samples to occlude it.
  2. For the second sample, its color is weighted by its alpha value and also by the remaining light that passed through the first sample (i.e., $(1-{\alpha }_1^{\prime })$).

2. Semantic Reconstruction

Obtain 2D semantic labels via α-blending based on the 3D semantic logit $s$:

$$\pi :\mathbf{S}=\sum _{i\in \mathcal{N}}\text{softmax}({\mathbf{s}}_i){\alpha }_i^{\prime }\prod _{j=1}^{i-1}(1-{\alpha }_j^{\prime }).$$

Note that we perform the softmax operation on 3D semantic logits $s_i$ *prior to $\ast \alpha$ blending, in contrast to most existing methods that apply softmax to 2D.

3. Optical Flow

$$\pi :\mathbf{F}=\sum _{i\in \mathcal{N}}{\mathbf{f}}_i{\alpha }_i^{\prime }\prod _{j=1}^{i-1}(1-{\alpha }_j^{\prime }),$$

where ${\mathbf{f}}_{t_1\to t_2}={\mu }_2^{\prime }-{\mu }_1^{\prime }$ and ${\mu }^{{}^{\prime }}$ is the projected center of Gaussian.

1.3.3 Loss function

$$\mathcal{L}={\mathcal{L}}_{\mathbf{I}}+{\lambda }_{\mathbf{S}}{\mathcal{L}}_{\mathbf{S}}+{\lambda }_{\mathbf{F}}{\mathcal{L}}_{\mathbf{F}}+{\lambda }_{\mathbf{t}}{\mathcal{L}}_{\mathbf{t}}+{\lambda }_{uni}{\mathcal{L}}_{uni}+{\lambda }_{reg}{\mathcal{L}}_{reg}$$

It is equal to image loss plus semantic loss plus optical flow plus Unicycle Model Losses.

1.3.4 Implementation Details

Pseudo-GTs:

We utilize InverseForm [5] to generate pseudo ground truth for semantic segmentation. For initializing the unicycle model, we employ a monocular-based method, QD-3DT [16], to acquire pseudo ground truth for 3D bounding boxes and tracking IDs at each training view. For optical flow, we use Unimatch [41] to obtain pseudo-ground truth.

1.4 My Takeaway and Self-thoughts

1. NSG + Gaussian Splatting
2. Unicycle model
3. Handle exposure
4. Render semantic maps and optical flow in Gaussian
5. Pseudo-GTs

Refer

[1] 13.1.2.3 A simple unicycle (uiuc.edu)

[2] HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting (xdimlab.github.io)