Method
-
Diffuse poses using a transformer
-
fix the first pose as identity always, make sure that transformer knows which is which using a special token
-
Represent a camera as \(\{q,t,\log(f)\}\)
-
In the usual diffusion formulation
\[p_\theta(x_{t-1} | x_t) = \mathcal{N}(x_{t-1} ; \sqrt{\alpha_t}\mathcal{D}_\theta(x_t, t), (1-\alpha_t) \mathbb{I})\]
Add a guidance term that steers poses using inverse distribution's score function
Inverse Distribution is
\[p(\mathtt{I} | x_t) = \prod_{i,j} p(I^i, I^j | x^i_t, x^j_t) \propto \prod_{i,j} exp(-e^{i,j})\]
\(e^{i,j}\) is the sampson epipolar error between \(I^i\) and \(I^j\) .
The forward distribution is \(p(x_t | \mathtt{I})\)
-
Diffusion outputs \(p(x_{t-1}|x_t)\) directly. The new mean with guidance is:
\[\hat{\mathcal{D}}_\theta(x_t, t, \mathtt{I}) = \mathcal{D}_\theta(x_t, t, \mathtt{I}) + s \nabla_{x_t} \log p(\mathtt{I} |x_t)\]
- We need to tune \(s\) to control guidance.