Novel view synthesis from a pair of images and camera
Two view image encoder
Given two images \(I\) and \(\tilde{I}\) , first take resnet50 and dino ViT-B/8 features, project/upsample and concatenate. Then run epipolar sampling and attention over the images.
Pixel-aligned Gaussian prediction
For each pixel, predict a discrete probability distribution over the depth with fixed number of bins. Sample a depth from this ( NeRF style) and use this to initialize the mean. Also predict covariance and spherical harmonics, use the probability of the sampled depth as the alpha (this makes it differentiable?). Predict other parameters using an MLP (outputs the probdist, delta, sigma directly, SH coeffs).
Training
Train for 300,000 steps at batch size 7, 80GB VRAM x 1 GPU