Diffusion model that takes \(N\) images and gives us both structure and pose, without the local-then-global approaches taken by most previous SfM methods. DiffusionSFM parameterizes structure and pose as pairs of origin and endpoints of rays in a global frame, then uses a transformer based denoising model to denoise these rays. Special steps are taken to address two challenges: ground truth depth data is sparse, and the scene scale is undetermined/scene is unbounded. Results are great, and the pipeline has a natural notion of uncertainty.
Prior Art
SfM is typically done in two steps:
-
A local/pairwise reasoning step
This might be something like handcrafted feature extraction and matching used in traditional methods (COLMAP), or something more neural/learned like Superpoint+ SuperGlue . This could also be a neural network encoding a scene incrementally like in AceZero. FlowMap also does something incremental.
-
A global optimization step
This could be something like a Bundle Adjustment routine, or learned optimization like in ba-net.
Single pass SfM methods exist:
-
Direct Regression
- SparsePose
-
Energy Minimization
- Relpose++
-
Diffusion
- PoseDiffusion
- RayDiffusion - closest to this method
Method
Given a set of sparse (2-8) images, DiffusionSfM predicts pose and structure in the global frame.
Parameterization
Given image \(I \in \mathbb{R}^{H \times W \times 3}\) , depth \(D \in \mathbb{R}^{H \times W}\) , intrinsics \(K \in \mathbb{R}^{3 \times 3}\) , camera to world transform \(T \in \mathbb{R}^{4 \times 4}\) or equivalently rotation \(R \in \mathrm{SO}(3)\) and translation \(t \in \mathbb{R}^{3}\) , each 2D pixel \(P_{i,j} = [u,v]\) has a corresponding ray from camera center \(C\) through the pixel, terminating at \(E_{i,j}\) :
\[ E_{i,j} = T^{-1} h(D_{i,j} K^{-1} [u,v,1]^{T})\]
where \(h : \mathbb{R}^3 \rightarrow \mathbb{P}^3\) transforms Euclidean points to Homogeneous. Each ray shares the same origin \(O_{i,j} = c = h(-R^{-1}t)\) . The origin and endpoint pair forms a ray \(S_{i,j}\) .
This is deliberately overparameterized – RayDiffusion suggests that this is good, and we can now use patch features like DINO for each ray.
DiffusionSfM
The diffusion network is a Diffusion Transformer (DiT), with the diffusion processes defined on the space \(\mathcal{S} = \mathrm{stack}(\{S^{(n)}\}_{n=1}^{N})\) :
\[ S_t = \sqrt{\bar{\alpha_t}} \mathcal{S_0} + \sqrt{1-\bar{\alpha_t}}\epsilon, \] where \(\epsilon \sim \mathcal{N}(0,1)\) , and \(\{\alpha_{t}\}_{t=1}^{T}\) defines a diffusion process with zero terminal SNR.
The DiT \(f_\theta\) is trained to predict \(\mathcal{S}_0\) at each timestep by minimizing the following loss:
\[L = \mathbb{E}_{t, \mathcal{S}_0, \epsilon} || \mathcal{S}_0 - f_\theta(\mathcal{S}_t, t, C) ||^2_2 .\]
Architecture
There are two models: a sparse variant called Diffuser, and a dense variant called DiffuserDPT.
Diffuser:
- Feature extraction using SpatialDino
- Diffusion Transformer (DiT) that outputs per-patch ray endpoints
DiffuserDPT
- Feature extraction using SpatialDino
- Diffusion Transformer (DiT) that outputs per-patch ray endpoints
- Dense prediction head that fuses DiT output with SpatialDino features (my read might be wrong here)
- Per pixel ray endpoints
Spatial Dino
Practical Considerations
Homogeneous Coordinates
Scene scale is underdetermined and unbounded, and Neural Networks, especially denoisers, need their inputs to be bounded.
So, the ray origins and endpoints are parameterized using homogeneous coordinates, with unit norm:
\[ (x,y,z) \rightarrow \frac{1}{w}(x,y,z,1) \]
\[ w = \sqrt{x^2+y^2+z^2+1} \]
Incomplete ground truth depths
CO3D and MegaDepth have sparse SfM ground truths, which means pixelwise depth is not defined for many pixels. We create a image-shaped mask that is true for pixels with valid GT depths, and use this in the following ways:
- Multiply noisy rays with the mask so that noisy rays aren't used in attention
- Concatenate mask M and (M*rays) channel-wise and input this to the DiT.
- Multiply pixel-wise loss terms with this mask.
During inference, we use an all true mask.
Sparse to dense training
First learn sparse DiT (raydiffusion style), and then learn a dense version initialized from the sparse one.
Experiments
Two variants: (1) CO3D, and (2) DUSt3r (habitat, CO3D, ScanNet++, ARKit Scenes, etc)
Training - CO3D Model
- Sparse Model: 4xH100, batch size 64, 400 000 iterations (2 days, ~100mil SGD iters)
- Dense model: 4xH100, batch size 48, 300 000 iterations (incl. warmup) (2 days, ~60mil SGD iters)
Results
Better sparse SfM all round.
Ablations show that both homogeneous parameterization and masks help.
Limits
- Pixel space diffusion might require larger model capacity than latent space diffusion
- Attention is square in number of images.