Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision

Diffusion is powerful, but we don't always have access to the training samples of the distribution we want to model. For example, say we want to model the distribution of a 3D scene, but we only have posed images as samples. This paper shows a way to model such distributions in the case where we have a differentiable forward function that maps a sample from the underlying distribution into the space where we have training samples.

Setup

Consider observations \((O^i_j, \phi^i_j)\) generated from underlying signals \(S_j\) according to a forward model \(O^i_j = \text{forward}(S_j, \phi^i_j)\) , where \(\phi^i_j\) are parameters of the forward model corresponding to observation \(O^i_j\) .

For example, \(O^i_j\) are image observations of 3D scenes \(S_j\) and \(\phi^i_j\) are the camera parameters. \(\text{forward()}\) is the rendering function.

Diffusion with Differentiable Forward Models

We want to model \(p(S | O^{ctxt}, \phi^{ctxt})\) over signals \(S\) given an observation \((O^{ctxt}, \phi^{ctxt})\) .

Split the observations into context and target observations. Run forward diffusion on the observations: \(q(O^{target}_t|O^{target}_{t-1}) = \mathcal{N}(O^{target}_t; \sqrt{1-\beta_t}O^{target}_{t-1}, \beta_t \mathrm{I})\) . In the reverse process, first predict an estimate of the signal \(S_{t-1}\) , then map it to an estimate of the denoised observations \(O^{trgt}_{t-1}\) using the forward model.

Training

Train the function \(\textrm{denoise}_\theta\) to minimize \(\lVert O^{target}_t - \text{forward}(\text{denoise}_\theta(O^{ctxt}, O^{trgt}_t; t, \phi^{ctxt}, \phi^{trgt})) \rVert\) .

Inverse Graphics

This is a specialization of the framework above, applied to the task of 3D reconstruction.

We use NeRF as the scene representation, such that \(S(p) = (\sigma, c)\) maps a 3D coordinate \(p\) to a color and density. Specifically, we use PixelNeRF.

Skipping some details, the PixelNeRF reconstruction of the scene is \(S(\cdot | \text{enc}(O^{\text{ctxt}}),\text{enc}(O^{\text{trgt}}))\) , dropping poses ( \(\phi\) terms) for brevity.

\(\text{enc}(O^{\text{trgt}}_t)\) is not informative enough, so we first generate a deterministic estimate of the scene given only the context view, and render a target view estimate. We concatenate this and the noisy target, and generate a 3D scene again, adding the features of the new deterministic view in another PixelNeRF step.

Single-image Motion Prediction

This specialization is cleaner than the one above and adheres more closely to the framework. The goal here is to train a model that lets us sample from all possible motions of pixels (optical flow) given a single static image. The underlying flow is unknown, and to train the model we use frame pairs that are temporally close together, sampled from videos.

The signal here is \(S(u) = (S_c(u), S_m(u))\) , which for each pixel \(u \in \mathbb{R}^2\) contains the color \(S_c(u) \in \mathbb{R}^3\) and 2D motion \(S_m(u) \in \mathbb{R}^2\) .

The forward operator is the `warp` function, implemented as a differentiable point splatting operation.

The forward model parameters are implicit: \(\phi_{\text{ctxt}} = 0\) and \(\phi_{\text{trgt}} = 1\) .

The denoiser is a 2D network that takes in \((O^{ctxt}, O^{trgt}_t, t)\) and predicts \(S_m\) .

We minimize a reconstruction loss on \(O^{ctxt}\) and an edge-aware smoothness regularization on \(S_m\) .

Branches