My notes here are from the first half of Adam Kosiorek's blog post What is wrong with VAEs?
Generative Modelling
We want to model the distribution \(p(x)\) over a state vector \(x \in \mathbb{R}^D\) . \(p(x)\) may be very complex, and so we want to factorize it as \(p(x) = p(x|z) p(z)\) , where \(z\) is another variable that explains the context that leads to \(x\) .
This turns \(p(x)\) , which we also call the marginal, into an infinite mixture model over \(z\) given by \[ p(x) = \int p(x|z) p(z) dz.\]
It would be useful to know, given a sample \(x\) , which choices of \(z\) explain it. That is, we would like to be able to evaluate the posterior \(p(z|x)\) .
\(p(z|x)\) could be highly a non-linear function, and \(D\) and \(d\) could be very large, which makes the marginal very difficult to evaluate. The posterior requires us to be able to do so, and the marginal in itself is interesting too (for generating novel samples, evaluating likelihood, and so on).
Approximations
We could use Monte Carlo sampling to evaluate \(p(x)\) , like so:
\[ p(x) \approx \frac{1}{M} \sum_{m=1}^{M} p(x | z^{m}), z^{m} \sim p(z) \]
The problem here is that \(M\) needs to be very large for this to work.
Learning the marginal directly
We could try to do something like
\[ \theta^* = \text{arg}\ \underset{\theta \in \Theta}{\text{max}} p_\theta(x), \]
but we would still need to evaluate \(p(x)\) exactly, which we can't do.
Variational Auto Encoders
VAEs have two parts:
-
\(p_\theta(x,z)\)
, which is the
generative model, made of two parts:
- \(p_\theta(x|z)\) , which is the decoder
- \(p(z)\) , which is the prior over latents
- \(q_\phi(z|x)\) , which is the encoder
The objective is to minimize the KL-divergence between \(q_\phi(z|x)\) and \(p_\theta(z|x)\) . We still dont know \(p_\theta(z|x)\) , but the KL-divergence takes us to an interesting place:
\[ KL\big[q_\phi(z|x) || p\theta(z|x)\big] = \mathbb{E}_{q_\theta(z|x)} \Bigl[\log \frac{q_\phi(x|z)}{p_\theta(z|x)}\Bigr] \]
\[ = \mathbb{E}_{q_\theta(z|x)} \Bigl[\log q_\phi(x|z) - \log p_\theta(z|x)\Bigr] \]
\[ = \mathbb{E}_{q_\theta(z|x)} \Bigl[\log q_\phi(x|z) - \log p_\theta(x,z)\Bigr] + \log p_\theta(x). \]
Rearranging terms, we get
\[ \log p_\theta(x) = KL\big[q_\phi(z|x) || p\theta(z|x)\big] - \mathbb{E}_{q_\theta(z|x)} \Bigl[\log \frac{q_\phi(x,z)}{p_\theta(z|x)}\Bigr] \]
Now, the KL term is at least 0, and so the expectation term is a lower bound on \(\log p_\theta(x)\) . So minimizing the expectation term improves the log likelihood of the sample \(x\) , or decreases the KL divergence (how?). This expectation is the Evidence Lower BOund (ELBO), which we'll label \(L(x; \theta, \phi)\) for now.
To train VAEs, we further approximate this expectation with a single sample of \(z\) given an \(x\) :
\[ L(x; \theta, \phi) \approx \log \frac{q_\phi(x,z)}{p_\theta(z|x)} \]
Now we optimize the parameters like so:
\[ \theta^*, \phi^* = \text{arg}\ \underset{\theta \in \Theta, \phi \in \Phi}{\text{max}} L(x; \theta, \phi)\] .
Adam's post is about a problem with VAEs, which is that they approximate the ELBO with a single sample of \(z\) . Instead, we could sample \(k\) plausible latents \(z^(i)\) , and approximate the ELBO with Importance Sampling, which has the following ramifications:
-
Pro: ELBO when calculated this way is tighter, and gets tighter with increasing \(k\)
-
Con: \(\nabla_\phi L\) vanishes with increasing \(k\) (its SNR \(\mathbb{E}[.]/\mathrm{Var}(.)\) vanishes too)
There are more clever Importance Sampling methods that make sure both θ and φ have well conditioned gradients for \(k \gt 1\) .