Structured Latents representation (SLAT)
Sparse voxel representation made of list of tuples (z, p) containing features and positions.
SLAT VAE:
-
Encoder (dino voxel -> SLAT)
create dino voxel using occupancy and multiview averaging of DINO features from spherically distributed inward facing cameras
Use Sparse VAE encoder to create SLATs
-
Decoder (SLAT -> 3dgs/rf/mesh)
For 3dgs, output K 3dgs per voxel, limit their position to within the voxel
3D Asset Generation
-
Conditioned on text or image, generate a sparse structure first using a flow transformer
Do this in a latent space because directly diffusing over a large voxel grid with binary values is hard
-
Denoise a SLAT, structured using the previously generated voxel grid, conditioned on the same image/text conditioning