【读论文】SUDS: Scalable Urban Dynamic Scenes

1. What

To extend NeRFs to dynamic large-scale urban scenes, this paper introduces two key innovations: (a) factorize the scene into three separate hash table data structures to encode static, dynamic, and far-field radiance fields, and (b) make use of unlabeled target signals consisting of RGB images(from N videos), sparse LiDAR(depth), off-the-shelf self-supervised 2D descriptors(DINO), and 2D optical flow.

2. Why

Challenge…

The largest dynamic NeRF built to date.

3. How

3.1 Inputs

RGB images from N videos + LiDAR depth measurements + 2D self-supervised pixel (DINO) + 2D optical flow without bounding boxes.

3.2 Representation

3.2.1 Scene composition and Hash tables

We have three branches to model the urban scenes:

1. A static branch containing non-moving topography consistent across videos.
2. A dynamic branch to disentangle video-specific objects [19,29,56], moving or otherwise.
3. A far-field environment map to represent far-away objects and the sky[41 Urban Neural Field].

As for the hash table, firstly, we have a given input coordinate $(\mathbf{x},\mathbf{d},\mathbf{t},\mathbf{vid})$. Then we need to find the surrounding voxels in each table, which we denote as ${\mathbf{v}}_{l,s},{\mathbf{v}}_{l,d},{\mathbf{v}}_{l,e}$.

The static branch makes use of 3D spatial voxels ${\mathbf{v}}_{l,s}$ while the dynamic branch makes use of 4D spacetime voxels ${\mathbf{v}}_{l,d}$ and the far-field branch makes use of 3D voxels ${\mathbf{v}}_{l,e}$(implemented via normalized 3D direction vectors) that index an environment map.

Then, we will compute hash indices ${\mathbf{i}}_{l,s}(\mathrm{or}{\mathbf{i}}_{l,d}\mathrm{or}{\mathbf{i}}_{l,e})$ for each corner with the following hash functions:

$$\begin{array}{rl}& {\mathbf{i}}_{l,s}=\mathrm{static}\mathrm{hash}(space({\mathbf{v}}_{l,s}))\\ & {\mathbf{i}}_{l,d}=\mathrm{dynamic}\mathrm{hash}(space({\mathbf{v}}_{l,d}),time({\mathbf{v}}_{l,d}),\mathbf{vid})\\ & {\mathbf{i}}_{l,e}=\mathrm{env}\mathrm{hash}(dir({\mathbf{v}}_{l,e}),\mathbf{vid})\end{array}$$

Recall the hash function is $h(\mathbf{x})=\left({⨁}_{i=1}^d{x}_i{\pi }_i\right)\mathrm{mod}T$.

Finally, we linearly interpolate features up to the nearest voxel vertices and obtain the feature vectors.

Notice: we add $\mathbf{vid}$ as an auxiliary input to the hash, but do not use it for interpolation (since averaging across distinct movers is unnatural). From this perspective, we leverage hashing to effectively index separate interpolating
functions for each video, without a linear growth in memory with the number of videos.

3.2.2 From branch to images

We will establish a relationship between the image and the feature vector obtained from the hash table:

1. Static branch

   $$\begin{array}{rl}& {\sigma }_s(\mathbf{x})\in \mathbb{R}\\ & {\mathbf{c}}_s(\mathbf{x},\mathbf{d},A_{vid}\mathcal{F}(t))\in {\mathbb{R}}^3.\end{array}$$

   We add a latent embedding $A_{vid}\mathcal{F}(t)$ consisting of a video-specific matrix $A_{vid}$ and a Fourier-encoded time index $\mathcal{F}(t)$.

2. Dynamic branch

   $$\begin{gathered} {\sigma }_d(\mathbf{x},\mathbf{t},\mathbf{vid})\in \mathbb{R} \\ {\rho }_d(\mathbf{x},\mathbf{t},\mathbf{vid})\in [0,1] \\ {\mathbf{c}}_d(\mathbf{x},\mathbf{t},\mathbf{vid},\mathbf{d})\in {\mathbb{R}}^3 \end{gathered}$$

   We find shadows to play a crucial role in the appearance of urban scenes, we explicitly model a shadow field of scalar values ${\rho }_d\in$ [0, 1], used to scale down the static color ${\mathbf{c}}_s$.

3. Far-field branch

   $${\sigma }_d(\mathbf{x},\mathbf{t},\mathbf{vid})\in \mathbb{R}$$

3.2.3 Rendering

Firstly, we derive a single density and radiance value for any position by computing the weighted sum of the static and dynamic components, combined with the pointwise shadow reduction:

$$\begin{array}{rl}\sigma (\mathbf{x},\mathbf{t},\mathbf{vid})& ={\sigma }_s(\mathbf{x})+{\sigma }_d(\mathbf{x},\mathbf{t},\mathbf{vid})\\ \mathbf{c}(\mathbf{x},\mathbf{t},\mathbf{vid},\mathbf{d})& =\frac{{\sigma }_s}{\sigma }(1-{\rho }_d){\mathbf{c}}_s(\mathbf{x},\mathbf{d},A_{vid}\mathcal{F}(t))\\ & +\frac{{\sigma }_d}{\sigma }{\mathbf{c}}_d(\mathbf{x},\mathbf{t},\mathbf{vid},\mathbf{d})\end{array}$$

Then similar to the $\alpha$-blending, we calculate the final color $\hat{C}$:

$$\begin{gathered} \begin{array}{rl}\hat{C}(\mathbf{r},\mathbf{t},\mathbf{vid})& ={\int }_0^{+\infty }T(t)\sigma (\mathbf{r}(t),\mathbf{t},\mathbf{vid})\mathbf{c}(\mathbf{r}(t),\mathbf{t},\mathbf{vid},\mathbf{d})dt\\ & +T(+\infty ){\mathbf{c}}_e(\mathbf{d},\mathbf{vid}),\end{array} \\ \text{where}T(t)=\exp \left(-{\int }_0^t\sigma (\mathbf{r}(s),\mathbf{t},\mathbf{vid})ds\right). \end{gathered}$$

3.2.4 Other supervision

1. Feature distillation

   We add a C-dimensional output head to each of our branches to predict the DINO feature and then compare it with the offline model.

   $$\begin{array}{rl}& {\Phi }_s(\mathbf{x})\in {\mathbb{R}}^C\\ & {\Phi }_d(\mathbf{x},\mathbf{t},\mathbf{vid})\in {\mathbb{R}}^C\\ & {\Phi }_e(\mathbf{d},\mathbf{vid})\in {\mathbb{R}}^C.\end{array}$$

   These are similar to the mapping of color. Firstly, $\mathbf{x}$ is passed to a grid encoding and then to a MLP to obtain the final output.
   But in the code, feature, flow, and color output use different grids, which makes their training independent.

               self.encoding_feature = tcnn.Encoding(
                   n_input_dims=5,
                   encoding_config={
                       'otype': 'SequentialGrid',
                       'n_levels': num_levels,
                       'n_features_per_level': features_per_level,
                       'log2_hashmap_size': log2_hashmap_size,
                       'base_resolution': base_resolution,
                       'include_static': False
                   },
               )
   

2. Scene flow

   $$s_{t^{\prime }\in [-1,1]}(\mathbf{x},\mathbf{t},\mathbf{vid})\in {\mathbb{R}}^3$$

3.3 Optimization

$$\begin{gathered} \mathcal{L}=\underset{\text{reconstruction losses}}{\underbrace{\left({\mathcal{L}}_c+{\lambda }_f{\mathcal{L}}_f+{\lambda }_d{\mathcal{L}}_d+{\lambda }_o{\mathcal{L}}_o\right)}}+\underset{\text{warping losses}}{\underbrace{\left({\mathcal{L}}_c^w+{\lambda }_f{\mathcal{L}}_f^w\right)}} \\ {\lambda }_{flo}\underset{\text{flow losses}}{\underbrace{\left({\mathcal{L}}_{cyc}+{\mathcal{L}}_{sm}+{\mathcal{L}}_{slo}\right)}}+\underset{\text{static-dynamic factorization}}{\underbrace{\left({\lambda }_e{\mathcal{L}}_e+{\lambda }_d{\mathcal{L}}_d\right)}}+{\lambda }_{\rho }{\mathcal{L}}_{\rho }. \end{gathered}$$

4. Experiment

4.1 City-Scale Reconstruction

It did not specify the dataset.

4.2 KITTI Benchmarks

KITTI + VKITTI (same as NSG)

4.3 Diagnostics

Flow-based warping is the single-most important input, while depth is the least crucial input.

Ref:

[1] SUDS: Scalable Urban Dynamic Scenes_suds论文解读-CSDN博客

[2] 【读论文】Instant Neural Graphics Primitives with a Multiresolution Hash Encoding-CSDN博客