DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion

Diffusion model that takes \(N\) images and gives us both structure and pose, without the local-then-global approaches taken by most previous SfM methods. DiffusionSFM parameterizes structure and pose as pairs of origin and endpoints of rays in a global frame, then uses a transformer based denoising model to denoise these rays. Special steps are taken to address two challenges: ground truth depth data is sparse, and the scene scale is undetermined/scene is unbounded. Results are great, and the pipeline has a natural notion of uncertainty.

Prior Art

SfM is typically done in two steps:

  1. A local/pairwise reasoning step

    This might be something like handcrafted feature extraction and matching used in traditional methods (COLMAP), or something more neural/learned like Superpoint+ SuperGlue . This could also be a neural network encoding a scene incrementally like in AceZero. FlowMap also does something incremental.

  2. A global optimization step

    This could be something like a Bundle Adjustment routine, or learned optimization like in ba-net.

Single pass SfM methods exist:

  1. Direct Regression
    • SparsePose
  2. Energy Minimization
    • Relpose++
  3. Diffusion

Method

Given a set of sparse (2-8) images, DiffusionSfM predicts pose and structure in the global frame.

Parameterization

Given image \(I \in \mathbb{R}^{H \times W \times 3}\) , depth \(D \in \mathbb{R}^{H \times W}\) , intrinsics \(K \in \mathbb{R}^{3 \times 3}\) , camera to world transform \(T \in \mathbb{R}^{4 \times 4}\) or equivalently rotation \(R \in \mathrm{SO}(3)\) and translation \(t \in \mathbb{R}^{3}\) , each 2D pixel \(P_{i,j} = [u,v]\) has a corresponding ray from camera center \(C\) through the pixel, terminating at \(E_{i,j}\) :

\[ E_{i,j} = T^{-1} h(D_{i,j} K^{-1} [u,v,1]^{T})\]

where \(h : \mathbb{R}^3 \rightarrow \mathbb{P}^3\) transforms Euclidean points to Homogeneous. Each ray shares the same origin \(O_{i,j} = c = h(-R^{-1}t)\) . The origin and endpoint pair forms a ray \(S_{i,j}\) .

This is deliberately overparameterized – RayDiffusion suggests that this is good, and we can now use patch features like DINO for each ray.

DiffusionSfM

The diffusion network is a Diffusion Transformer (DiT), with the diffusion processes defined on the space \(\mathcal{S} = \mathrm{stack}(\{S^{(n)}\}_{n=1}^{N})\) :

\[ S_t = \sqrt{\bar{\alpha_t}} \mathcal{S_0} + \sqrt{1-\bar{\alpha_t}}\epsilon, \] where \(\epsilon \sim \mathcal{N}(0,1)\) , and \(\{\alpha_{t}\}_{t=1}^{T}\) defines a diffusion process with zero terminal SNR.

The DiT \(f_\theta\) is trained to predict \(\mathcal{S}_0\) at each timestep by minimizing the following loss:

\[L = \mathbb{E}_{t, \mathcal{S}_0, \epsilon} || \mathcal{S}_0 - f_\theta(\mathcal{S}_t, t, C) ||^2_2 .\]

Architecture

There are two models: a sparse variant called Diffuser, and a dense variant called DiffuserDPT.

Diffuser:

DiffuserDPT

Spatial Dino

Practical Considerations

Homogeneous Coordinates

Scene scale is underdetermined and unbounded, and Neural Networks, especially denoisers, need their inputs to be bounded.

So, the ray origins and endpoints are parameterized using homogeneous coordinates, with unit norm:

\[ (x,y,z) \rightarrow \frac{1}{w}(x,y,z,1) \]

\[ w = \sqrt{x^2+y^2+z^2+1} \]

Incomplete ground truth depths

CO3D and MegaDepth have sparse SfM ground truths, which means pixelwise depth is not defined for many pixels. We create a image-shaped mask that is true for pixels with valid GT depths, and use this in the following ways:

During inference, we use an all true mask.

Sparse to dense training

First learn sparse DiT (raydiffusion style), and then learn a dense version initialized from the sparse one.

Experiments

Two variants: (1) CO3D, and (2) DUSt3r (habitat, CO3D, ScanNet++, ARKit Scenes, etc)

Training - CO3D Model

Results

Better sparse SfM all round.

Ablations show that both homogeneous parameterization and masks help.

Limits