pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction

Novel view synthesis from a pair of images and camera

Two view image encoder

Given two images \(I\) and \(\tilde{I}\) , first take resnet50 and dino ViT-B/8 features, project/upsample and concatenate. Then run epipolar sampling and attention over the images.

Pixel-aligned Gaussian prediction

For each pixel, predict a discrete probability distribution over the depth with fixed number of bins. Sample a depth from this ( NeRF style) and use this to initialize the mean. Also predict covariance and spherical harmonics, use the probability of the sampled depth as the alpha (this makes it differentiable?). Predict other parameters using an MLP (outputs the probdist, delta, sigma directly, SH coeffs).

Training

Train for 300,000 steps at batch size 7, 80GB VRAM x 1 GPU