Forwards
Projection
Project 3D mean \(\mu\) and covariance \(\Sigma\) into 2D mean and covar \(\mu'\) and \(\Sigma'\)
Projection of a point onto a camera has multiple stages:
-
Transform point from the world frame into the camera frame.
\[ t = T_{\text{cw}} [ \mu 1 ]^\top \]
-
Project the point onto the image plane \[ t' = P t \]
-
Compute pixel coordinates \[ \mu' = ... \]
So the mean is projected as …
For covariance, we use covariance propagation:
σ' = J R S Rt Jt, J from taylor series
Paramterize Sigma as such: Sigma = RSStRt
Alpha composition
Bin gaussians into 16x16 tiles, sort by depth
compute the axis aligned bb around 99% conf ellipse of each covariance and include it in a bin if its bb intersects with the tile
C = ∑ c n a n Tn, Tn = ∏ (1-am) for m < n
alpha = o n . exp(-sigma n )
sigma n = 1/2 (delta.T sigma.ing delta)
compute T n online
Backwards
Alpha composition
d(C i (k))/dc n = a n Tn
compute T n-1 online (store T N for each pixel in the forward pass)
d(C i (k))/d an = C n (k).T n - (S n (k))/(1-alpha n ), S n = sum(c m a m T m )
This comes from d T n / d alpha m = T n / 1-α m if m > n, zero otherwise