FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Views

Predict camera poses and gaussians given a set of images.

Neural Pose Predictor

Predict coarse poses (q,t) for all images using a small decoder-only transformer

Camera centric geometry estimation

Predict pointmaps for each view using a DPT transformer

Global geometry projector

Project each local pointmaps to a globally-consistent frame using another DPT transformer

Predict Gaussians

Regress mean, scale, rotation, opacity and SH coefficients, normalize mean and scale with the gt and prediction scales.

Training

Train the pipeline to minimize GS reprojection loss, pose loss, geometry loss (like dust3r) and depth loss (between monocular depth estimation and rendered depth).

Results

Results upto 25 images, their method OOMs beyond 25. Trained on 8 images per reconstruction. 14 days x 64 a800 gpus.

Something I don't understand: are their gaussians arranged in a grid?