Structured 3D Latents for Scalable and Versatile 3D Generation

Structured Latents representation (SLAT)

Sparse voxel representation made of list of tuples (z, p) containing features and positions.

SLAT VAE:

  1. Encoder (dino voxel -> SLAT)

    create dino voxel using occupancy and multiview averaging of DINO features from spherically distributed inward facing cameras

    Use Sparse VAE encoder to create SLATs

  2. Decoder (SLAT -> 3dgs/rf/mesh)

    For 3dgs, output K 3dgs per voxel, limit their position to within the voxel

3D Asset Generation

  1. Conditioned on text or image, generate a sparse structure first using a flow transformer

    Do this in a latent space because directly diffusing over a large voxel grid with binary values is hard

  2. Denoise a SLAT, structured using the previously generated voxel grid, conditioned on the same image/text conditioning