[Notes] How covariance matrix determind 3DGS and 2DGS

The covariance matrix ($\text{cov}$) defines the shape and orientation of the Gaussian in both 3D and 2D. The covariance matrix plays a key role in determining the geometric properties of the Gaussian splat in space. Let’s break down the key aspects of the 3D and 2D covariance matrices and compare how they influence the resulting shape and orientation of the splat.

1. Covariance in 3D ($\text{cov3d}$)

The 3D covariance matrix describes the shape and orientation of the Gaussian in 3D space. It’s a symmetric 3x3 matrix, which can be written as:
$$\text{cov3d}=\left[\begin{array}{ccc}{\sigma }_{xx}& {\sigma }_{xy}& {\sigma }_{xz}\\ {\sigma }_{xy}& {\sigma }_{yy}& {\sigma }_{yz}\\ {\sigma }_{xz}& {\sigma }_{yz}& {\sigma }_{zz}\end{array}\right]$$

• Diagonal elements (${\sigma }_{xx},{\sigma }_{yy},{\sigma }_{zz}$): These represent the variances (or square of the standard deviations) along the principal axes $x$, $y$, and $z$ of the 3D space. They define the size or extent of the Gaussian along these axes. In other words, they control how the Gaussian stretches or contracts along these directions.

• Off-diagonal elements (${\sigma }_{xy},{\sigma }_{xz},{\sigma }_{yz}$): These represent the correlations between different axes. They define how much the Gaussian is tilted or sheared in space, which controls the orientation of the ellipsoid that the Gaussian forms in 3D.

Influence of 3D Covariance Matrix:

1. Shape: The eigenvalues of the 3D covariance matrix determine the shape of the Gaussian ellipsoid. Large eigenvalues correspond to a more spread-out Gaussian along the corresponding eigenvector, while small eigenvalues correspond to a more compact Gaussian.

2. Orientation: The eigenvectors of the covariance matrix define the orientation of the Gaussian ellipsoid in 3D space. The off-diagonal terms (${\sigma }_{xy},{\sigma }_{xz},{\sigma }_{yz}$) control how the Gaussian is rotated relative to the axes.

2. Covariance in 2D ($\text{cov2d}$)

The 2D covariance matrix describes the shape and orientation of the Gaussian after it has been projected from 3D into 2D space (e.g., onto an image plane). It’s a symmetric 2x2 matrix, which can be written as:
$$\text{cov2d}=\left[\begin{array}{cc}{\sigma }_{xx}^{\prime }& {\sigma }_{xy}^{\prime }\\ {\sigma }_{xy}^{\prime }& {\sigma }_{yy}^{\prime }\end{array}\right]$$
Where:

• Diagonal elements (${\sigma }_{xx}^{\prime }$, ${\sigma }_{yy}^{\prime }$): These represent the variances (or squares of the standard deviations) along the principal axes (now in 2D, typically the image plane). They define the width and height of the Gaussian in 2D.

• Off-diagonal element (${\sigma }_{xy}^{\prime }$): This represents the correlation between the $x$ and $y$ axes in 2D. If non-zero, it indicates that the 2D Gaussian is tilted or rotated with respect to the $x$- and $y$-axes.

Influence of 2D Covariance Matrix:

1. Shape: Similar to the 3D case, the eigenvalues of the 2D covariance matrix determine the shape of the Gaussian ellipse in 2D. This ellipse corresponds to the 2D projection of the original 3D ellipsoid.

2. Orientation: The off-diagonal term (${\sigma }_{xy}^{\prime }$) indicates the tilt or rotation of the 2D Gaussian. If ${\sigma }_{xy}^{\prime }=0$, the Gaussian is aligned with the $x$- and $y$-axes. If ${\sigma }_{xy}^{\prime }\ne 0$, the Gaussian is rotated.

---

Comparing cov3d and cov2d

The key difference between the 3D and 2D covariance matrices is that cov3d defines the shape and orientation of the Gaussian in the full 3D space, while cov2d defines the shape and orientation of the Gaussian after it has been projected into 2D. The projection process involves several transformations (such as applying the view matrix, perspective divide, and Jacobian), which change the shape and orientation of the Gaussian.

How the 3D Covariance Matrix Affects the 2D Projection:

1. Projection of the 3D Ellipsoid:

   • The 3D Gaussian is represented by an ellipsoid defined by the covariance matrix $\text{cov3d}$. When projected onto the 2D image plane, this ellipsoid becomes an ellipse.
   • The shape of the 2D ellipse is determined by how the 3D ellipsoid is aligned relative to the camera. If the ellipsoid is aligned with the viewing direction, the 2D ellipse will be circular. If not, the ellipse will be elongated in certain directions.
   • The orientation of the 2D ellipse depends on the orientation of the 3D ellipsoid. This orientation is encoded in the off-diagonal terms of $\text{cov3d}$, which describe the correlations between the $x$, $y$, and $z$ axes. After projection, these correlations are transformed into the 2D off-diagonal term ${\sigma }_{xy}^{\prime }$.

2. Effects of Camera and Perspective:

   • The view matrix and the projection matrix (which includes the focal lengths $fx$ and $fy$ affect how the 3D covariance matrix is transformed into 2D. These transformations are applied as part of the projection process.
   • The perspective division (using $1/z$) in the projection step introduces non-linearity, meaning that points farther from the camera are projected differently than points closer to the camera. This also affects the size of the projected Gaussian in 2D space.
   • The Jacobian matrix $J$, which is involved in the transformation from 3D to 2D, plays a critical role in determining how the 3D covariance matrix is modified during the projection. Specifically, it adjusts the scale of the Gaussian based on the depth ($z$) and the camera parameters ($fx$, $fy$).

---

Specific Parameters and Their Influence

Parameter in 3D Covariance	Influence in 3D	Influence in 2D (after projection)
${\sigma }_{xx}$ (Variance along $x$)	Controls the spread along the $x$-axis in 3D space.	Influences the width of the Gaussian in the image plane, especially if the 3D Gaussian is aligned with the camera.
${\sigma }_{yy}$ (Variance along $y$)	Controls the spread along the $y$-axis in 3D space.	Influences the height of the Gaussian in the image plane. If the 3D Gaussian is oriented along the viewing direction, this will affect the vertical spread of the projected Gaussian.
${\sigma }_{zz}$ (Variance along $z$)	Controls the spread along the $z$-axis (depth) in 3D space.	Influences the overall size of the Gaussian in 2D after perspective projection. If the object is far from the camera, the Gaussian will appear smaller in the image plane.
${\sigma }_{xy}$ (Correlation between $x$ and $y$)	Introduces a tilt or shear in the 3D ellipsoid.	After projection, this results in a rotated or tilted Gaussian in the image plane.
${\sigma }_{xz}$ (Correlation between $x$ and $z$)	Introduces a tilt or shear between the $x$- and $z$-axes in 3D.	After projection, this affects the apparent size and orientation of the Gaussian in 2D. It can lead to a stretched Gaussian if the point is far from the camera.
${\sigma }_{yz}$ (Correlation between $y$ and $z$)	Introduces a tilt or shear between the $y$- and $z$-axes in 3D.	Affects the vertical size of the Gaussian in the image plane, especially if the point is far from the camera.

---

Summary of Key Differences and Influences

• The 3D covariance matrix ($\text{cov3d}$) defines the full 3D shape and orientation of the Gaussian in space. It controls the spread along each axis (via the diagonal elements) and the orientation or tilt (via the off-diagonal elements).

• The 2D covariance matrix ($\text{cov2d}$) is the result of projecting the 3D Gaussian onto the 2D image plane. It defines the width, height, and tilt of the Gaussian in the image space. The 3D covariance matrix is transformed through the camera projection matrix and perspective division, leading to the 2D covariance matrix.

• Diagonal elements in both cov3d and cov2d control the size of the Gaussian along each axis, while off-diagonal elements control the orientation or tilt of the Gaussian.