PolyDiffuse: Polygonal Shape Reconstruction via Guided Set Diffusion Models

DDPMs cannot learn sets very well

Traditional DDPM has two limitations for learning sets:

  1. Sets are defined upto permutation: a set with N elements has N! different but equivalent representations, which makes denoising factorially ambiguous.
  2. Denoising process needs to reach a single solution for a reconstruction task, that is, it must output the order that our ground truth data follows. But at higher noise levels, all tokens look just about the same – it isn't clear where each token needs to go because they all look the same.

Guided Set Diffusion Model (GS-DM)

In the forward process, instead of adding noise from the same distribution to all elements, GS-DM adds noise dependent on mean and stddev dependent on the timestep, the element, and its index in the data.

Additionally, factorize stddev into \(sigma(x_0, t, i) = \sqrt{1 - \bar{alpha}_t} C(x_0, i)\) .

In the reverse process, we don't have access to \(x_0\) , so we use a proposal generator to estimate \(x_0\) , and use it to produce the guidance noise.