Diffusion is powerful, but we don't always have access to the training samples of the distribution we want to model. For example, say we want to model the distribution of a 3D scene, but we only have posed images as samples. This paper shows a way to model such distributions in the case where we have a differentiable forward function that maps a sample from the underlying distribution into the space where we have training samples.
Setup
Consider observations \((O^i_j, \phi^i_j)\) generated from underlying signals \(S_j\) according to a forward model \(O^i_j = \text{forward}(S_j, \phi^i_j)\) , where \(\phi^i_j\) are parameters of the forward model corresponding to observation \(O^i_j\) .
For example, \(O^i_j\) are image observations of 3D scenes \(S_j\) and \(\phi^i_j\) are the camera parameters. \(\text{forward()}\) is the rendering function.
Diffusion with Differentiable Forward Models
We want to model \(p(S | O^{ctxt}, \phi^{ctxt})\) over signals \(S\) given an observation \((O^{ctxt}, \phi^{ctxt})\) .
Split the observations into context and target observations. Run forward diffusion on the observations: \(q(O^{target}_t|O^{target}_{t-1}) = \mathcal{N}(O^{target}_t; \sqrt{1-\beta_t}O^{target}_{t-1}, \beta_t \mathrm{I})\) . In the reverse process, first predict an estimate of the signal \(S_{t-1}\) , then map it to an estimate of the denoised observations \(O^{trgt}_{t-1}\) using the forward model.
Training
Train the function \(\textrm{denoise}_\theta\) to minimize \(\lVert O^{target}_t - \text{forward}(\text{denoise}_\theta(O^{ctxt}, O^{trgt}_t; t, \phi^{ctxt}, \phi^{trgt})) \rVert\) .
Inverse Graphics
This is a specialization of the framework above, applied to the task of 3D reconstruction.
We use NeRF as the scene representation, such that \(S(p) = (\sigma, c)\) maps a 3D coordinate \(p\) to a color and density. Specifically, we use PixelNeRF.
Skipping some details, the PixelNeRF reconstruction of the scene is \(S(\cdot | \text{enc}(O^{\text{ctxt}}),\text{enc}(O^{\text{trgt}}))\) , dropping poses ( \(\phi\) terms) for brevity.
\(\text{enc}(O^{\text{trgt}}_t)\) is not informative enough, so we first generate a deterministic estimate of the scene given only the context view, and render a target view estimate. We concatenate this and the noisy target, and generate a 3D scene again, adding the features of the new deterministic view in another PixelNeRF step.
Single-image Motion Prediction
This specialization is cleaner than the one above and adheres more closely to the framework. The goal here is to train a model that lets us sample from all possible motions of pixels (optical flow) given a single static image. The underlying flow is unknown, and to train the model we use frame pairs that are temporally close together, sampled from videos.
The signal here is \(S(u) = (S_c(u), S_m(u))\) , which for each pixel \(u \in \mathbb{R}^2\) contains the color \(S_c(u) \in \mathbb{R}^3\) and 2D motion \(S_m(u) \in \mathbb{R}^2\) .
The forward operator is the `warp` function, implemented as a differentiable point splatting operation.
The forward model parameters are implicit: \(\phi_{\text{ctxt}} = 0\) and \(\phi_{\text{trgt}} = 1\) .
The denoiser is a 2D network that takes in \((O^{ctxt}, O^{trgt}_t, t)\) and predicts \(S_m\) .
We minimize a reconstruction loss on \(O^{ctxt}\) and an edge-aware smoothness regularization on \(S_m\) .
Branches
- Solving Inverse Problems with FLAIR - flow-based priors
- Spatial Reasoning with Denoising Models - flow solves sudoku?!