Score networks are inaccurate in regions (space and time) without training data. Langevin dynamics might not converge correctly when a sampling trajectory encounters those regions.
The paper considers noise scales that \(\left{\sigma_i\right}^{L}_{i=1}\) that decrease monotonically.
Annealed LD: multiple noise scales and time steps, which are different.
There is a choice of whether to denoise \(x_T\) or not, which we can do with the Tweedie's formula, and it helps boost FID.
TODO Read the original NCSN paper
This paper shows how to choose in theoritically motivated ways:
- The set of noise scales
- The way that the score network incorporates the information of \(\sigma\)
- Step size parameter \(\epsilon\)
- The number of sampling steps per noise scale \(T\) .
Choosing Noise Scales
When original data is in \([0, 1]\) , NCSN recommends choosing a geometric sequence of noise scales, with 10 scales, from 1 to 0.01.
Questions - Is the first scale being 1, length 10 and geometric progression ideal?
Initial Noise Scale
Considering that as the number of samples \(N \rightarrow \infty\) , \(p_{\text{data}}(x)\) is given by sum over delta functions at each sample, normalized by \(N\) . This same distribution, when perturbed with the standard normal (why do this?) is given by a mixture model, where each component is a standard normal distribution centered at a sample.
Then, they go on to say:
For generating diverse samples regardless of initialization, we naturally expect that Langevin dynamics can explore any component \(p(i)(x)\) when initialized from any other component \(p(j)(x)\) , where \(i \ne j\) ."
What this means is that it should be possible for Langevin dynamics starting at a sample to end up at any other sample, which I think in practice makes it so that what your initial noise does not matter, since it can transition to some other place easily.
With some maths, we see that \(\sigma_1\) has to be numerically comparable to the maximum pariwise distances of data.
Other noise scales
In higher dimensions, Gaussians look less like fuzzy balls and more like bubbles.
We need the samples from a noise level to be likely at the lower noise level too - otherwise the network will not have seen enough (lower noise level, higher noise level sample) during training. Choosing this overlap to be some constant in terms of likelihood (there's \(\log_2C\) bits of likelihood difference between the 3-sigma belts of the radial distributions of subsequent noise scales) results in noise scales forming a geometric progression.
Ideally, we have have a large noise scale, they form a very smooth progression, and \(C \approx 1\) . But having too many noise scales makes sampling costly.
The original NSCN used \(L=10\) for 32 \(\times\) 32 images, for which \(C=0\) upto numerical precision. This paper recommends \(C \approx 0.5\) , for which an appropriate geometric progression common ratio might be found.
Incorporating noise information into the network
In the limit of a single data sample, the expectation of score is \(\approx \sqrt{D}/\sigma\) . Empirically, the NCSN paper notes that predicted score has norm inversely proportional to \(\sigma\) . Because \(\sigma\) varies dramatically in terms of orders of magnitude, learning this might be difficult for neural network, which prefer their inputs and outputs to be well-scaled.
So, this paper recommends parameterizing NSCNs with \(\s_\theta(x, \sigma) = s_\theta(x)/\sigma\) .
Configuring Annealed Langevin Dynamics
TODO - my understanding here is too superficial to write anything
EMA
Plotting FID vs training step shows general improvement, but it is not very stable. Also, the color of the samples shift in tint with training progress. To fix these, use an EMA weighted version of the network weights for inference.
Results
Using all of these techniques improves FID from 26 to 10 in CelebA and 25 to 11 in CIFAR-10 Unconditional. FID is more stable with training progress, tint shift goes away, and NCSN loss reduces faster.