Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency

ReSample - solve general inverse problems with pretrain LDMs

For direct inverse problems, the usual way of doing it is to go:

The DDIM reverse step is \(x_{t-1} = x_0(x_t) + dx_{t-1} + \eta.\)

For inverse problems, these are the Diffusion Posterior Sampling (DPS) steps:

  1. Take a DDIM like step \[x'_{t-1} = x_0(x_t) + s_\theta(x_{t-1}) + \eta,\]
  2. Take a step in the direction that minimizes inconsistency \[x_{t-1} = x'_{t-1} - \zeta \nabla_{x_t}|| y - A(x_0(x_t)) ||_2^2.\]

We could do the same for latent diffusion models, using these steps (call this Latent-DPS):

  1. Take a DDIM like step on the latent variable \[z'_{t-1} = z_0(z_t) + s_\theta(z_{t-1}) + \eta,\]
  2. Take a step in the direction that minimizes inconsistency \[z_{t-1} = z'_{t-1} - \zeta \nabla_{z_t}|| y - A(D(z_0(z_t))) ||_2^2.\]

The main ideas in this paper are:

  1. Instead of taking a single consistency step, solve for the most consistent sample.
  2. Solve using gradient descent, initialized at \(\hat{z}_0(z_{t-1})\) .
  3. Do this for some steps only (this is not very useful in the high noise regime, so use it in the last 2/3 steps in reverse diffusion)