UVGS: Reimagining Unstructured 3D Gaussian Splatting using UV Mapping

Three main contributions:

uvgs

uvgs is an image like representation of 3DGS that does not suffer from permutation invariance using spherical mapping.

Consider a sphere that bounds a 3DGS object, centered at the object center. Project all Gaussians on to the sphere, and create a 14-channel UV map, where each pixel contains the 14-dimensional feature of a Gaussian projected inside it. With center (σ ∈ ℝ 3 \), quaternion (r ∈ ℝ 4 \), scale (s ∈ ℝ 3 \), opacity \(o \in \mathbb{R}^3\) , and color (c ∈ ℝ 3 \), the UV map is an \(M \times N\) , 14-channel image:

\[ U[\phi, \theta, :] = [ \sigma_i, r_i, s_i, o_i, c_i ] \in \mathbb{R}^{14}.\]

If more than one gaussians project to the same pixel on \(U\) , resolve the collision using one of these two ways:

  1. Dynamic Gaussian Selection: Select the Gaussian with the highest opacity.
  2. Multiple Layers: Use \(k * 14\) channels and store the \(k\) -most opaque Gaussians along the pixel's ray.

Super-uvgs

Super-uvgs is a 3-channel representation that can be used with off the shelf, pretrained image models.

The authors train a branched CNN that learns a mapping from 14 channels uv map to a 3 channel uv map. They find that the 3 channel uv map is compatible with pretrained, off-the-shelf models (the autoencoders in LDMs for example)

Uses

The authors train LDMs over super-uvgs and show applications such as unconditional generation, inpainting (claim that theirs is the first method to inpaint in 3dgs), and more.