Most of my (incomplete) notes here are based on Lilian Weng's blog at https://lilianweng.github.io/posts/2021-07-11-diffusion-models .
The goal is to sample new points from the real data distribution \( \mathrm{x}_0 \sim q(\mathrm{x}) \) . We don't know what it is, but we have some samples from the distribution in our dataset.
We'll first define two processes:
Forward Process
Given a data point x, we add a small amount of Gaussian noise to the sample, for T total steps. The noise added at step \(t\) has a mean of 0 and variance defined by a schedule \(\{\beta_t \in (0,1)\}_{t=1}^{T}\) .
This gives us
\[ \mathrm{x}_t = \sqrt{1 -\beta_t}\cdot \mathrm{x}_{t-1} + \beta_t \epsilon,\]
where \(\epsilon \sim \mathcal{N}(0, 1).\)
Recursively expanding, we get this closed form expression for \(q(\mathrm{x}_t |\mathrm{x}_0 )\)
\[ \mathrm{x}_t = \sqrt{\bar\alpha_t}\cdot \mathrm{x}_0 + \sqrt{1- \bar\alpha_t}\cdot \epsilon,\]
where \(\alpha_t = 1 - \beta_t\) and \(\bar{\alpha}_t = \prod_{i=1}^t \alpha_i\) . As \(T \rightarrow \infty, \mathrm{x}_t \rightarrow \epsilon\) , since \(\beta_t\) and thus \(\alpha_t\) are all in \((0,1)\) , leading to \(\bar{\alpha}_T \rightarrow 0\) .
Reverse Process
If we had \(q(\mathrm{x}_{t-1}|\mathrm{x}_t)\) , we could sample \(q(\mathrm{x})\) by ancestral sampling from \(q(\mathrm{x}_T)\) . But we don't.
Note that if \(\beta_t\) is small, \(q(\mathrm{x}_{t-1}|\mathrm{x}_t)\) will be a Gaussian (how?).
\(q(\mathrm{x}_{t-1}|\mathrm{x}_t, \mathrm{x}_0)\) is tractable though. Using Bayes' rule, expanding Gaussian forms, separating the \(\mathrm{x}_{t-1}\) and \(\mathrm{x}_{t-1}^2\) terms, we get its mean and variance (refer to Lilian's blog for the complete steps):
\[ \begin{aligned} \tilde{\beta}_t &= \frac{\sqrt{\alpha_t}(1 - \bar{\alpha}_{t-1})}{1 - \bar{\alpha}_t} \mathbf{x}_t + \frac{\sqrt{\bar{\alpha}_{t-1}}\beta_t}{1 - \bar{\alpha}_t} \mathbf{x}_0\\ \tilde{\mu}_t &= {\frac{1}{\sqrt{\alpha_t}} \Big( \mathbf{x}_t - \frac{1 - \alpha_t}{\sqrt{1 - \bar{\alpha}_t}} \epsilon_t \Big)} \end{aligned} \]