Manifold Sampling for Differentiable Uncertainty in Radiance Fields

Reasoning about uncertainty in GS is difficult.

We want a method for uncertainty estimation that is:

There are two main ways to model uncertainty: Variational methods and Laplace's approximation

Assuming a Gaussian distribution over parameters is probably a bad idea because, for example, colors are bounded values, and an uninformed prior concentrated around some color is not natural.

The goal is to learn a radiance field \(F_\theta\ : \left(x, d, \xi \right) \to \left(\sigma, c\right)\) from training data in the form of per-pixel triplets \(\tau = \{r_i, \xi_i, C_i \}^{N_\tau}_{i=1}\) . \(\xi\) is an optional vector of auxiliary parameters, for example, a scalar time dimension for dynamic reconstruction, or vector of illumination conditions for relighting.

Notice two generalizations: \(\xi\) accomodates general scene configurations, and we don't require multiple training images.

This work uses 3D Gaussian Splatting as the radiance field, defined by \(\theta \in \mathbb{R}^{d_\theta}\) , the set of all trainable parameters, with \(d_\theta\) often in the millions. Assume we have a function \(C(r, \xi) \to \mathbb{R}^3\) mapping a ray to a color.

Stochastic Radiance Field

Consider a distribution of radiance fields \(p(F_\theta)\) realized by modelling the joint distribution \(p(\theta)\) of all parameters. Sampling \(\theta^\star \sim p(\theta) \) realizes a specific radiance field.

Our objective is to optimize \( p(F_\theta | \tau) \) by minimizing rendering errors for training views while concurrently maximizing uncertainty for novel views. We want the most spread out posterior that still each realization still explains the training data.

We approximate \(p(\theta)\) using a continuous uniform distribution over a compact hyper-volume \(V \subset \mathbb{R}^{d_\theta}\)

\[ p(\theta) = \begin{cases} \frac{1}{|V|} & \theta \in V, \\ 0 & \text{otherwise} \end{cases} \]

How we construct the hypervolume is discussed later. Now, we look at the training loss, per-pixel uncertainty in rendering.

For optimal V, the training loss is:

\[ L = \sum^{N_\tau}_{i=1} \int_{\theta^\star \in V} { || C_{\theta*\star}(r_i, \xi_i) - C_i ||_1 d \theta^{\star}} - \lambda |V| \]

Here, \(\lambda\) is a scalar, and the second term encourages large hypervolumes.

The rendered color and uncertainties are the mean and standard deviations of the rendering across all realizations:

\[ \bar{C}(r, \xi) = \int_{\theta^\star \in V} C_{\theta^\star}(r, \xi)d\theta^\star \]

\[ U(I(\xi)) = \sum_{r \in I} \int_{\theta^\star \in V} || C_{\theta^\star}(r, \xi) - \bar{C}_{\theta^\star}(r, \xi) ||^2 d\theta^\star \]

We cannot solve these integrals analytically since \(C\) is a complicated function and \(\theta\) is a very high dimensional variable. We use Monte Carlo integration because it scales to high dimensions in principle. Using \(M\) samples, the Monte Carlo estimators for above equations are:

\[ L_{\text{MC}} = \sum^{N_\tau}_{i=1} \frac{1}{M} \sum_{\theta^\star \sim V} { || C_{\theta*\star}(r_i, \xi_i) - C_i ||_1 d \theta^{\star}} - \lambda |V| \]

\[ \bar{C_{MC}}(r, \xi) =\frac{1}{M} \sum_{\theta^\star \sim V} C_{\theta^\star}(r, \xi)d\theta^\star \]

\[ U_{MC}(I(\xi)) = \sum_{r \in I} \frac{1}{M} \sum_{\theta^\star \sim V} || C_{\theta^\star}(r, \xi) - \bar{C}_{\theta^\star}(r, \xi) ||^2 d\theta^\star \]

Constructing \(V\)

We need a volume \(V \in \mathbb{R}^{d_\theta}\) that captures the distribution of trainable parameters \(\theta\) . Ideally, \(V\) should not be too restricted in shape, and sampling, rendering, and representing \(V\) should be space and compute efficient.

Consider a linear model \(G\) of the form \(G(z) = \bar{\theta} + Bz\) , where \(\bar{\theta} \in \mathbb{R}^{d_\theta}\) is the mean of the distribution, and \(z \sim \mathcal{U}([-1,1]^{d_\theta})\) . \(B \in \mathbb{R}^{d_\theta \times d_\theta}\) is the generating matrix, and it defines the shape of \(V\) as a parallelotope. The covariance of the emerging distribution of parameter samples is given by \(\Sigma = BB^T \in \mathbb{R}^{d_\theta \times d_\theta}\) .

Using full \(B\) is memory and compute intensive. Some alternatives are:

  1. Independent components in \(\theta\) : \(B\) and \(\Sigma\) become diagonal matrices and \(V\) becomes an axis-aligned hyper-rectangle.
  2. Independent blocks in \(\theta\) : \(B\) and \(\Sigma\) become block-diagonal matrices and \(V\) becomes a parallelotope with a restricted distribution of admissible slopes.
  3. Low-rank approximation \(\hat{B} \in \mathbb{R}^{d_{\theta} \times k}, k \ll d_\theta\) : \(\Sigma\) becomes a rank- \(k\) matrix, \(V\) becomes a \(k\) -dimensional linear manifold in ℝ d θ .

The authors find that 1 and 2 are too restrictive, and 2 needs us to choose beforehand what parameters to group together, and these groups need to be small to retain tractability. 3 works well, even with \(k=2\) :

  1. It is compact, requiring only \((k+1) \times d_\theta\) parameters.
  2. The generation model \(G(\hat{z}) = \bar{\theta} + \hat{B}\hat{z}\) is efficient to compute.
  3. The low dimensionality of \(V\) makes the Monte Carlo estimators for loss, color, and uncertainty highly effective and noise-free.

Special attention must be paid to the paramterization of \(\hat{B}\) . We need the columns in \(\hat{B}\) to be linearly independent to allow the manifold to have an effective dimensionality close to \(k\) . The authors choose to pass \(\hat{B}\) through a relu, and multiply each element with a random but fixed sign, which essentially guarantees independence.

The training objective now becomes

\[L_{\text{manif}} = { \sum^{N_\tau}_{i=1} \frac{1}{M} { \sum_{z \sim \mathcal{U}([-1,1]^{k})} { || C_{G(\hat{z})}(r_i, \xi_i) - C_i ||_1 d \theta^{\star} } } } - \lambda ||\hat{B}||_1 \]

with similar derivations for color and uncertainty.

The authors use the Sobol sequence to sample from \(\mathcal{U}\) . They set \(\lambda = 1\) every 10 training iterations, and 0 otherwise. They use \(M=2\) to render the uncertainty map.