Interpreting and Improving Diffusion Models from an Optimization Perspective

Denoising is approximate projection onto data manifold

\(x_t\) - scaled pred noise is a good approximation of projection of x t onto data manifold when \(\sqrt(n) \sigma\) is a good approximation of distance from x to the manifold.

Denoising \(\sim\) \(\underset{x_t}{\mathrm{min}} \ \mathrm{dist}(x_t, \mathrm{\mathcal{K}})\)

Denoising algorithms are approximately gradient descent on the squared distance from \(x_t\) to the manifold \(\mathcal{K}\) .

For example, DDIM is (under assumptions) approximately gradient descent using learning rate \(1 - \sigma_{t-1} / \sigma_t\) .