3D Gaussian Splatting as Markov Chain Monte Carlo

A simpler optimization strategy for 3DGS that:

An MCMC objective for 3DGS

Consider rasterizing \(N\) sorted gaussians with colors \(c_i\) , opacities \(o_i\) , location \(\mu_i\) , covariance \(\mathbf{\Sigma}\) from pose \(\theta\) . The color of a pixel \(\mathrm{x\) is given by:

\[ C(\mathrm{x}) = \sum_{i=1}^{N} c_i \alpha_i(\mathrm{x}) \left[ \prod_{j=1}^{i-1} (1-\alpha_j{\mathrm{x}}) \right] ,\]

with

\[ \alpha_i(\mathrm{x}) = o_i \exp \Biggl( -\frac{1}{2}(\mathrm{x}- R_{\theta}(\mu_i))^\mathrm{T}R_\theta(\Sigma_i)^{-1}(\mathrm{x}- R_\theta(\mu_i)) \Biggr) \]

where \(R_\theta\) is the camera projection operator.

The 3DGS reconstruction loss is:

\[ \mathcal{L}_{\text{orig}} = (1 - \lambda_{\text{D-SSIM}}) \mathcal{L}_1 + \lambda_{\text{D-SSIM}} \mathcal{L}_{\text{D-SSIM}}. \]

For a single gaussian \(g\) , the SGD update rule is:

\[ g \leftarrow g - \lambda_{\text{lr}} \cdot \nabla_g \mathbb{E}_{I \sim \mathcal{I}} \left[ \mathcal{L}_{\text{total}} (g; I) \right].\]

A typical SGLD update rule would be:

\[ g \leftarrow g - a \cdot \nabla_g \log \mathcal{P}(g) + b \cdot \epsilon, \]

where \(\mathcal{P}\) is the data-dependent probability density function and \(\epsilon\) is the exploration noise distribution. SGLD reduces to SGD when \(\lambda_{\text{lr}} = -a\) and \(b=0\) .

We design a new update rule for Gaussians:

\[ g \leftarrow g - \lambda_{\text{lr}} \cdot \nabla_g \mathbb{E}_{I \sim \mathcal{I}} \left[ \mathcal{L}_{\text{total}} (g; I) \right] + \lambda_{\text{noise}} \cdot \epsilon\]

In practice, \(\epsilon\) needs to be balanced by the gradient term, otherwise the update rule reduces to random update. The support of a Gaussian might reduce irrecoverably, which breaks ergodicity that MCMC requires. We add noise to \(\mu_i\) only, experiments show that adding noise to other parameters hurts performance. We also add less noise to well-behaved, high opacity Gaussians since their gradients are strong enough for good guidance. We shape the noise like so:

\[ \epsilon = \lambda_{\text{lr}} \cdot \sigma(-k(o-t)) \cdot \mathbf{\Sigma} \cdot \eta ,\]

where \(\eta \sim \mathcal{N}(\mathbf{0},\mathbf{I})\)

Relocation / splitting gaussians

We want to split a Gaussian into \(N\) such that the MCMC distribution does not change.

Consider a sole Gaussian which results in the color \(C^{\text{old}}(\mathrm{x}) \) in pixel \(\mathrm{x}\) . We first set \(c^\text{new} = c^\text{old} \) . If we set \(C^{\text{new}}(\mu) = C^{\text{old}}(\mu) \) , we get

\[ (1-o^{\text{new}})^N = 1-o^{\text{old}}. \]

This alone does not preserve rasterization and thus \(P_{\text{old}}\) . For that, we require \(\int_{-\infty}^{\infty} {C^{\text{new}}(\mathrm{x}) d\mathrm{x}} = \int_{-\infty}^{\infty} {C^{\text{old}}(\mathrm{x}) d\mathrm{x}}\) . This does not have a simple analytical solution. If this is to hold true, the integrals must be the same for an arbitrary 1-D slice in x that passes through \(\mu\) . This constraint involves single variable integrals that are tractible, and the result can be generalized to the full integrals. The paper derives this, but I don't understand how the jump from 1-D to n-D works.

\[ \Sigma^{\text{new}} = (o^{\text{old}})^2 \cdot \Biggl( \sum_{i=1}^{N}{ \sum_{k=0}^{i-1} { \biggl( \biggl( \binom{i-1}{k} \frac{ (-1)^k (o^{\text{new}})^2)^{k+1} } {\sqrt{k+1}} \biggr) \biggl) } } \Biggr)^{-2} \cdot \mathbf{\Sigma^{\text{old}}} \]

We sample the new means using the old Gaussian's covariance.

Ideas