Three main contributions:
uvgs
uvgs is an image like representation of 3DGS that does not suffer from permutation invariance using spherical mapping.
Consider a sphere that bounds a 3DGS object, centered at the object center. Project all Gaussians on to the sphere, and create a 14-channel UV map, where each pixel contains the 14-dimensional feature of a Gaussian projected inside it. With center (σ ∈ ℝ 3 \), quaternion (r ∈ ℝ 4 \), scale (s ∈ ℝ 3 \), opacity \(o \in \mathbb{R}^3\) , and color (c ∈ ℝ 3 \), the UV map is an \(M \times N\) , 14-channel image:
\[ U[\phi, \theta, :] = [ \sigma_i, r_i, s_i, o_i, c_i ] \in \mathbb{R}^{14}.\]
If more than one gaussians project to the same pixel on \(U\) , resolve the collision using one of these two ways:
- Dynamic Gaussian Selection: Select the Gaussian with the highest opacity.
- Multiple Layers: Use \(k * 14\) channels and store the \(k\) -most opaque Gaussians along the pixel's ray.
Super-uvgs
Super-uvgs is a 3-channel representation that can be used with off the shelf, pretrained image models.
The authors train a branched CNN that learns a mapping from 14 channels uv map to a 3 channel uv map. They find that the 3 channel uv map is compatible with pretrained, off-the-shelf models (the autoencoders in LDMs for example)
Uses
The authors train LDMs over super-uvgs and show applications such as unconditional generation, inpainting (claim that theirs is the first method to inpaint in 3dgs), and more.