Predict camera poses and gaussians given a set of images.
Neural Pose Predictor
Predict coarse poses (q,t) for all images using a small decoder-only transformer
Camera centric geometry estimation
Predict pointmaps for each view using a DPT transformer
Global geometry projector
Project each local pointmaps to a globally-consistent frame using another DPT transformer
Predict Gaussians
Regress mean, scale, rotation, opacity and SH coefficients, normalize mean and scale with the gt and prediction scales.
Training
Train the pipeline to minimize GS reprojection loss, pose loss, geometry loss (like dust3r) and depth loss (between monocular depth estimation and rendered depth).
Results
Results upto 25 images, their method OOMs beyond 25. Trained on 8 images per reconstruction. 14 days x 64 a800 gpus.
Something I don't understand: are their gaussians arranged in a grid?