ScenarioDreamer
-
LDM generates a scene \(\mathcal{I}_F\) represented as a set \({O,M}\) , sets of objects and map structure respectively.
-
\(O\) is a set with \(N_o\) objects, each with an 8-dimensional vector (2 each for position, speed, heading angle (sine and cosine), dimensions (lengh,width)), and object class.
-
\(M\) consists of a set \(L\) of \(N_l\) centerlines, where each centerline is a 20-long sequence of 2d positions. \(M\) also consists of a matrix \(A \in \mathbb{R}^{N_l \times N_l \times 4}\) , which contains the four adjacencies for successor, predessor, left and right neighbor connections.
-
The generator must also be able to inpaint (generate consistent BEV-tiles that go beyond an existing tile).
Autoencoder
-
Transformer based encoder \(\mathcal{E}_\phi\) and decoder \(\mathcal{D}_\gamma\) that directly work on vectorized scene elements.
-
Per-vector MLP for each vector first.
-
Then, \(N_E\) factorized attention blocks (each block has lane-to-lane, lane-to-object, and object-to-object attention).
-
Map each object and lane embedding to latent dimensions \(K_o\) and \(K_l\) respectively, and a \(\beta\) -VAE on top.
-
No object-to-lane attention so that lane latents do not depend on the object features, which allows lane-conditioned object generation at inference.
LDM
-
Sample scene elements (sample \(N_o\) and \(N_l\) from training set statistics).
-
AdaLN-Zero conditioning which enables variable latent sizes.
-
Factorized denoiser blocks (object-to-lane, lane-to-lane, lane-to-object, object-to-object), object and lane tokens can be of different sizes this way.
-
DDPM, noise prediction formulation.
-
Permutation ambiguity - set structured data brings challenges because at sufficient noise levels, there is not enough structure for the model to reliably infer permutation of the ground truth signal towards which it should regress. So they set a canonical ordering on the latents depending on the bounding box coordinates, with a form of collision resolution.