Cameras as Rays: Pose Estimation via Ray Diffusion

The goal is to recover cameras (intrinsics and extrinsics) from a sparse set of images \(\{I_1, \dots, I_N\}\) .

Summary:

Camera Representation

They hypothesize that a compact representation \((K [R|t])\) is harder to learn than over-parameterized rays.

Their camera representation is: \[ R = \{r_1, \dots, r_m\}, \] where each ray \(r_i \in \mathbb{R}^6\) is parameterized using Plücker coordinates:

\[ r = \left< d, m \right> \in \mathbb{R}^6,\] where \(m = p \times d\in \mathbb{R}^3\) is the moment vector, which is agnostic to the specific point on the ray used to compute it.

To convert cameras to ray bundle:

\[ d = R^T K^{-1} u, \hspace{3em} m = (-R^T t) \times d\]

For the reverse:

\[ c = \mathop{\mathrm{argmin}}\limits_{p \in \mathbb{R}^3} \sum_{\left<d,m\right> \in R} || p \times d - m ||^2\] \[ P = \mathop{\mathrm{argmin}}\limits_{||H|| = 1} \sum_{i=1}^{m} || H d_i \times u_i ||\]

\(P\) is a homography which we can compute using DLT, and we can recover \(R\) and \(K\) using RQ-decomposition. Then, \(t = -R^T c\) .

Ray Regressor

Ray Denoiser

Training and Implementation Details

Metrics

Rotation error

Translation error