Mathematical Supplement for the \ \(\textbackslash texttt\gsplat\\\) Library

Forwards

Projection

Project 3D mean \(\mu\) and covariance \(\Sigma\) into 2D mean and covar \(\mu'\) and \(\Sigma'\)

Projection of a point onto a camera has multiple stages:

  1. Transform point from the world frame into the camera frame.

    \[ t = T_{\text{cw}} [ \mu 1 ]^\top \]

  2. Project the point onto the image plane \[ t' = P t \]

  3. Compute pixel coordinates \[ \mu' = ... \]

So the mean is projected as …

For covariance, we use covariance propagation:

σ' = J R S Rt Jt, J from taylor series

Paramterize Sigma as such: Sigma = RSStRt

Alpha composition

Bin gaussians into 16x16 tiles, sort by depth

compute the axis aligned bb around 99% conf ellipse of each covariance and include it in a bin if its bb intersects with the tile

C = ∑ c n a n Tn, Tn = ∏ (1-am) for m < n

alpha = o n . exp(-sigma n )

sigma n = 1/2 (delta.T sigma.ing delta)

compute T n online

Backwards

Alpha composition

d(C i (k))/dc n = a n Tn

compute T n-1 online (store T N for each pixel in the forward pass)

d(C i (k))/d an = C n (k).T n - (S n (k))/(1-alpha n ), S n = sum(c m a m T m )

This comes from d T n / d alpha m = T n / 1-α m if m > n, zero otherwise