The goal is to recover cameras (intrinsics and extrinsics) from a sparse set of images \(\{I_1, \dots, I_N\}\) .
Summary:
- Distributed, over-parameterized camera representation - bundle of rays
- Two models: (1) camera regressor that outperforms previous sparse pose estimation methods, and (2) camera denoiser that works even better
Camera Representation
They hypothesize that a compact representation \((K [R|t])\) is harder to learn than over-parameterized rays.
Their camera representation is: \[ R = \{r_1, \dots, r_m\}, \] where each ray \(r_i \in \mathbb{R}^6\) is parameterized using Plücker coordinates:
\[ r = \left< d, m \right> \in \mathbb{R}^6,\] where \(m = p \times d\in \mathbb{R}^3\) is the moment vector, which is agnostic to the specific point on the ray used to compute it.
To convert cameras to ray bundle:
\[ d = R^T K^{-1} u, \hspace{3em} m = (-R^T t) \times d\]
For the reverse:
\[ c = \mathop{\mathrm{argmin}}\limits_{p \in \mathbb{R}^3} \sum_{\left<d,m\right> \in R} || p \times d - m ||^2\] \[ P = \mathop{\mathrm{argmin}}\limits_{||H|| = 1} \sum_{i=1}^{m} || H d_i \times u_i ||\]
\(P\) is a homography which we can compute using DLT, and we can recover \(R\) and \(K\) using RQ-decomposition. Then, \(t = -R^T c\) .
Ray Regressor
-
Compute rays over a uniform \(p \times p\) grid such that \(m = p^2\)
-
Compute spatial image features of each patch, use each patch feature as a token
-
Concatenate the pixel coordinates of a patch (in uncropped NDC) to the corresponding patch feature token
-
Using a Diffusion Transformer, predict rays:
\[ \hat{R}_{i=1}^N = f_{\text{regress}} (\{f_i, u_i \}_{i=1}^{N \cdot p^2}) \]
Ray Denoiser
-
Use noisy pixel coordinates to initialize rays ( \(u_i\) are no longer uniformly spaced).
\[ \hat{R}_{i=1}^N = f_{\text{diffuse}} (\{f_i, u_i, r_{i,t} \}_{i=1}^{N \cdot p^2}, t) \]
Training and Implementation Details
- Both models are trained using an \(L_2\) loss over ray errors.
-
To fix
\(\text{Sim}(3)\)
gauge:
- Place the world origin at the point closest to all camera axes, which is a useful inductive bias for object-centric captures
- Set the first camera at identity rotation and unit distance away from world origin, fixing scale and orientation.
- Use a frozen DINOv2 (S/14) feature extractor.
- Use 100 timesteps for diffusion.
- Use a DiT with 16 transformer blocks as the architecture for both denoiser and regressor (set t=100 always).
- Training takes 2 and 4 days on 8xA6000 respectively for the regressor and denoiser.
-
Empirical tricks:
- Remove stochasticity in DDPM inference
- Stop reverse diffusion early (earlier steps help select among distinct plausible modes, later steps yield samples around these modes (what does this mean?), accuracy metrics (could?) prefer modes)
Metrics
Rotation error
- Find all pairwise relative rotations for both predicted and ground truth cameras
- Find error between ground truth and predicted relative rotations
- Report proportion of errors within 15 degrees
Translation error
- Sim3 align predicted poses with ground truth poses
- Report proportion of camera center translations that lie within 0.1 * scene scale
- Scene scale is the distance from the origin to the farthest camera center