【读论文】SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes

SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes

1. What

What kind of thing is this article going to do (from the abstract and conclusion, try to summarize it in one sentence)

Inputting a monocular dynamic video, this paper uses sparse control points to drive Gaussian. Each control point has 6 DoF which is time-varying and can be predicted by a MLP. This method can enable dynamic view synthesis and motion editing but still has some limitations in inaccurate poses or intense movements.

2. Why

Under what conditions or needs this research plan was proposed (Intro), what problems/deficiencies should be solved at the core, what others have done, and what are the innovation points? (From Introduction and related work)

Maybe contain Background, Question, Others, Innovation:

Nerf-based methods struggle with low rendering qualities, speeds, and high memory usage. Existing 3D-GS only applies to static scenes. An intuitive method [47] involves learning a flow vector for each 3D Gaussian, but it incurs a significant time cost for training and inference(The author of 47 is a co-author of this).

Related work:

• Dynamic NeRF

• Dynamic Gaussian Splatting

• 3D Deformation and Editing

  This part is relatively unfamiliar. It introduces the traditional editing methods in graphics which focus on preserving the geometric details of 3D objects during the deformation process, containing some descriptors like Laplacian coordinates, Poisson equation, and cage-based approaches.

  Recently, there have been other approaches that aim to edit the scene geometry learned from 2D images. This paper belongs to this class.

3. How

3.1 Sparse Control Points

We will first introduce the definition of control points, which is the core concept used in this article.

There are a set of sparse control points P = { ( p i ∈ R 3 , o i ∈ R + ) } , i ∈ { 1 , 2 , ⋯   , N p } \mathcal{P}=\{(p_{i}\in\mathbb{R}^{3},o_{i}\in\mathbb{R}^{+})\},i\in \{1,2,\cdots,N_{p}\} P={(pi​∈R3,oi​∈R+)},i∈{1,2,⋯,Np​}. And $o_i$ is a learnable radius parameter that controls how the impact of a control point on a Gaussian.

Meanwhile, for each control point $k$, we learn time-varying 6 DoF transformations $[R_i^t|T_i^t]\in \mathbf{SE}(3)$ , consisting of a local frame rotation matrix $R_i^t\in \mathbf{SO}(3)$ and a translation vector $T_i^t\in {\mathbb{R}}^3$. But instead of directly optimizing the transformation parameters, we employ an MLP $\Psi$ to learn a time-varying transformation field:

$$\Psi :(p_i,t)\to (R_i^t,T_i^t).$$

3.2 Dynamic Scene Rendering

After having some control points, we need to found the connection between it with the Gaussian.

We use the k-nearest neighbor (KNN) search to obtain its K(= 4) neighboring control points denoted as { p k ∣ k ∈ N j } \{p_{k}|k\in\mathcal{N}_{j}\} {pk​∣k∈Nj​}. Then define its weight as:

$$w_{jk}=\frac{{\hat{w}}_{jk}}{\sum _{k\in {\mathcal{N}}_j}{\hat{w}}_{jk}},\text{where}{\hat{w}}_{jk}=\exp (-\frac{d_{jk}^2}{2o_k^2}),$$

where $d_{jk}$ is the distance between the center of Gaussian $G_j$ and the neighboring control point $p_k$, and $o_k$ is the learned radius parameter of neighboring control points $p_k.$

Then learn from the ideas of Linear Blend Skinning (LBS), which tells us each vertex in the model is assigned a weight for each bone indicating how much influence that bone has over the vertex’s position. When a bone moves, each vertex moves according to a weighted average of the transformations (rotations and translations) of all the bones that influence it. So we can adjust the Gaussian by:

$$\begin{array}{rl}{\mu }_j^t& =\sum _{k\in {\mathcal{N}}_j}{w}_{jk}\left(R_k^t({\mu }_j-p_k)+p_k+T_k^t\right)\\ q_j^t& =(\sum _{k\in {\mathcal{N}}_j}{w}_{jk}{r}_k^t)\otimes q_j,\end{array}$$

The new rotation is the weighted average of the rotations of the neighboring control points, represented as quaternions $r_k^t$ (the same meaning as $R_k^t$, just different in the mathematical form). This average quaternion is then multiplied by the original rotation quaternion of the Gaussian using the quaternion product, which combines the two rotations in a way that is appropriate for 3D rotations.

3.3 Optimization

We have known how to control Gaussian by the control points. Then we will introduce two small strategies in optimization.

1. ARAP Loss

A loss function was introduced from the paper “As-rigid-as-possible surface modeling”, which helps maintain rigidity.

Firstly, it defines points’ trajectories $p_i^{traj}$ in the scene motion as:

$$p_i^{\text{traj}}=\frac{1}{N_t}{p}_i^{t_1}\oplus p_i^{t_2}\oplus \cdots \oplus p_i^{t_{N_t}},$$

where $p_i^t$ represents the position of point $i$ in time $t$.

Then, a local neighborhood for each control point is determined via ball queries, which means finding all control points within a predefined radius to define a local area of influence. To calculate ${\mathcal{L}}_{\mathrm{arap}}$, we randomly sample two time steps $t_1$ and $t_2$. For each point $p_k$ within the radius $(i.e.k\in {\mathcal{N}}_{\mathrm{c}i})$, its transformed locations with learned translation parameters $T_k^{t_1}$ and $T_k^{t_2}$ are: $p_k^{t_1}=p_k+T_k^{t_1}$ and $p{k}^{t_2}=p_k+T_k^{t_2}$, thus the new rotation matrix ${\hat{R}}_i$ can be estimated as:

$${\hat{R}}_i=\arg \underset{R\in \mathbf{SO}(3)}{\min }\sum _{k\in {\mathcal{N}}_{c_i}}{w}_{ik}||(p_i^{t_1}-p_k^{t_1})-R(p_i^{t_2}-p_k^{t_2})|{|}^2.$$

Finally, ${\mathcal{L}}_{\mathrm{arap}}$ can be calculated as:

$${\mathcal{L}}_{\mathrm{arap}}(p_i,t_1,t_2)=\sum _{k\in {\mathcal{N}}_{c_i}}{w}_{ik}||(p_i^{t_1}-p_k^{t_1})-{\hat{R}}_i(p_i^{t_2}-p_k^{t_2})|{|}^2.$$

2. Adaptive Control Points

Another strategy, very similar to the Adaptive Control of Gaussian.

• Prune: calculate its overall impact $W_i={\sum }_{j\in \tilde{N}i}{w}_{ji}$ on the set of Gaussians $j\in {\tilde{\mathcal{N}}}_i$ whose K nearest neighbors include $p_i.$ Then, we prune $p_i$ if $W_i$ is close to zero, indicating little contribution to the motion of 3D Gaussians.

• Clone: calculate the summation of the Gaussian gradient norm as:

  $$g_i=\sum _{j\in {\tilde{\mathcal{N}}}_i}{\tilde{w}}_j\Vert \frac{d\mathcal{L}}{d{\mu }_j}{\Vert }_2^2,\mathrm{where}{\tilde{w}}_j=\frac{w_{ji}}{\sum _{j\in {\tilde{\mathcal{N}}}_k}{w}_{ji}}.$$

  A large $g_k$ indicates poor reconstruction. And add a new point $p_k^{{}^{\prime }}$ as

  $$p_k^{\prime }=\sum _{j\in {\tilde{\mathcal{N}}}_k}{\tilde{w}}_i{\mu }_j;{\sigma }_k^{\prime }={\sigma }_k.$$

Now, we can utilize this pipeline to understand every process.

3.4 Motion Editing

Given a set of user-defined handle points { h l ∈ R 3 ∣ l ∈ H ⊂ { 1 , 2 , ⋯   , N p } } \{h_l\in\mathbb{R}^3\mid l\in\mathcal{H}\subset \{1,2,\cdots,N_p\}\} {hl​∈R3∣l∈H⊂{1,2,⋯,Np​}}, the control graph $P^{\prime }$ can be deformed by minimizing the APAR energy formulated as:

$$E({\mathcal{P}}^{\prime })=\sum _{i=1}^{N_p}\sum _{j\in {\mathcal{N}}_i}{w}_{ij}||(p_i^{\prime }-p_j^{\prime })-{\hat{R}}_i(p_i-p_j)|{|}^2,$$

with the fixed position condition $p_l^{\prime }=h_l\mathrm{for}l\in \mathcal{H}.$

4. Self-thoughts

1. Only use one camera, if we use six cameras like AD, how to combine them?
2. The edit of Gaussian is with the help of physical rigid body motion.
3. ARAP can be replaced by ARAPReg.
4. Only suitable for small objects with meaningful control points. Can we use only a few control points like human posture representation?