ReSample - solve general inverse problems with pretrain LDMs
For direct inverse problems, the usual way of doing it is to go:
The DDIM reverse step is \(x_{t-1} = x_0(x_t) + dx_{t-1} + \eta.\)
For inverse problems, these are the Diffusion Posterior Sampling (DPS) steps:
- Take a DDIM like step \[x'_{t-1} = x_0(x_t) + s_\theta(x_{t-1}) + \eta,\]
- Take a step in the direction that minimizes inconsistency \[x_{t-1} = x'_{t-1} - \zeta \nabla_{x_t}|| y - A(x_0(x_t)) ||_2^2.\]
We could do the same for latent diffusion models, using these steps (call this Latent-DPS):
- Take a DDIM like step on the latent variable \[z'_{t-1} = z_0(z_t) + s_\theta(z_{t-1}) + \eta,\]
- Take a step in the direction that minimizes inconsistency \[z_{t-1} = z'_{t-1} - \zeta \nabla_{z_t}|| y - A(D(z_0(z_t))) ||_2^2.\]
The main ideas in this paper are:
- Instead of taking a single consistency step, solve for the most consistent sample.
- Solve using gradient descent, initialized at \(\hat{z}_0(z_{t-1})\) .
- Do this for some steps only (this is not very useful in the high noise regime, so use it in the last 2/3 steps in reverse diffusion)