3D Gaussian Splatting for Real-Time Radiance Field Rendering

Use gaussians! SOTA quality! fast renders!

Scene Representation

The scene is made of Gaussians in world space, defined by the mean \(\mu\) and a 3x3 covariance matrix \(\Sigma\) . These Gaussians do not have to be normalized pdfs. Each Gaussian has an alpha value, which is a single scalar that we can use for alpha blending when we render the scene later. This is a very Lagrangian/explicit scene representation.

Rendering

To render, we need to project the individual Gaussians onto the image plane. We can directly project the mean using the projection function. For the covariance, we need to linearize the projection function as \(P = J W\) , where \(W\) is the viewing transformation and \(J\) is the Jacobian of the affine approximation of the projective transformation.

Only the first 2x2 block of the projected covariance matters: we do not care about the covariance terms involving depth axis in the view-space.

Parameterizing Gaussians

Covariance matrices have to be positive semi-definite, and a naive optimization method like gradient descent does not in general obey this constraint. We can work around this (find the closest PSD matrix after each iteration), but a better way is to reparameterize \(\Sigma\) in terms of a scalar matrix \(S\) and a rotation matrix \(R\) . \(S\) has 3 parameters on the diagonal, which we write as a vector \(s\) . \(R\) can again be parameterized as a quaternion \(q\) . Both \(s\) and \(q\) can easily be converted to their respective matrix forms \(S\) and \(R\) .

\[\Sigma = R S S^\intercal R^\intercal\]

Quaternions when used as rotations are over-parameterised and so at any iteration the rotation might not actually be valid. So, we normalize quaternions after every step.

Rendering

We could in theory render the splats in the same way we would render any other volumetric representations, using the same rendering pipeline used in NeRFs. But because splats are discrete, sparse representations, we can do better.

3DGS uses a tile-based rendering approach which lends itself well to parallelization on the GPU. We first form 16x16 tiles of the image, then cull all Gaussians against the view frustrum and the boundaries of each of the tiles. Gaussians that span more than one tile need to be copied so that one gaussian appears in each tile.

Then, each gaussian is assigned a 64-bit tag, the first 32-bit is the viewspace depth, and the second is the ID of the tile to which the gaussian belongs. We sort the global gaussian list on this tag (basically on depth) using GPU Radix sort.

Everything from now on happens per tile. We find the first and last splat that belongs to the tile. We loop through these splats front to back, and then accumulate color using alpha blending. When the alpha saturates for all pixels (the rays have gone through enough "stuff" to not see anything farther), we are done rendering this tile.

When all the tiles are done, the frame has been rendered.

Loss

The loss compares the rendered frame with the target frame, and has two parts.

  1. L1 photometric loss
  2. D-SSIM D-SSIM (Data-Structural SIMilarity index) is a similarity index which is closer to how we perceive similariy in images than L1 or L2.

\[L = (1-\lambda) L_1 + \lambda L_{D-SSIM}\]

Controlling scene density

Optimizing the loss using stochastic gradient descent does not result in good scene geometry (?). We need to regularize the Gaussians in the scene, and there's multiple ways that 3DGS does this:

  1. Densification Every 100 iterations, remove Gaussians that are transparent ( \(\alpha < \eta_\alpha\) ).
  2. In under reconstructed regions: Copy the gaussian, and nudge the copy along the positional gradient.
  3. In over reconstructed regions: Sample two gaussians, divide their variances by 1.6, which is determined experimentally.
  4. Set \(\alpha\) close to zero every 3000 iterations, and let the first step remove unneeded geometry.
  5. Periodically remove gaussians that are very large in worldspace, or that have very large footprint in viewspace.

Implementation details

3DGS is implemented in PyTorch, with custom CUDA kernels for rasterization. For radix sort, it uses NVIDIA CUB (which is part of CCCL now). The optimization begins at 4x smaller image resolution, and we upsample once each after 250 and 500 iterations. 4 bands of spherical harmonics coefficients are used, starting with the zero-order component only at first, and one new band is introduced every 1000 iterations.

Observations / Questions

My ideas

Implementations

forward pass: {'projection': 0.13312000036239624, 'isect': 1.4192639589309692, 'rasterization': 1.9660799503326416} backward: 13.06163215637207

branches