Semi-Dense Mapping
- Estimate a SD inv-depth map for current frame
- Use this to estimate camera pose of the next frame
- Filter depth maps and propagate to the keyframe
- Keep per-pixel inverse depth hypothesis
- Notation: \(d_i\) is the inverse depth map of a pixel \(x_i\)
Stereo to find depth information
The idea is to use small baseline stereo where we can
Depth map update algorithm
-
Find pixels with high disparity accuracy
- They identify 3 criteria for this
- For each pixel, select a suitable reference frame, and do a 1-dim disparity search on the epiline
- Use priors where possible to make this fast (reduce the disarity search space)
- Fuse observations into the current depth map
Reference Frame Selection
-
The ideal reference frame
- Maximizes stereo accuracy
- Keep disparity search space and observation angle sufficiently small
-
Heuristic
- Use the oldest frame the pixel was observed in as the reference for that pixel if the disparity and observation angle do not exceed a threshold
- If disparity search fails, increase pixel's age
Stereo matching
- exhaustive search and then sub-pixel localize
- if a prior depth hypothesis is available, limit search to \(d \pm 2 \sigma_d\)
- use full epiline otherwise
-
use SSD error over 5 equidistant points on the epiline within the
search space
- increases robustness in the HF regions
- 4/5 of these interpolates are reused
Uncertainty Estimation
- Some maths to come to an upper bound on depth variance \[\sigma_{d_{obs-SSD}} \le \alpha^2 ( \min(\sigma^2_{e, \pi}) + \min(\sigma^2_{\lambda I}))\]
- The first term is the error due to view point and projection function
- the second term is the error due to image grads (regions with no image grad have higher matching error)
- Select good pixels based on this bound on variance
- Stereo match these points only
Depth map fusion
- Inputs: two depth maps
- Posterior = prior * observation, Bayesian update
- Initialize with observation if no prior exists (would be the same as initializing everything with zero information priors)
Summary
- Calculate upper bound on matching error
- select promising pixels for matching
- choose reference frame for each of those pixels
- match (1-D search on epiline)
- integrate these measurements into the current keyframe's depthmap
Depth map propagation
\[d_1(d_0) = (d^{-1} - t_z)^{-1}\]
- \(t_z\) is the camera translation along the optical axis
- error for this transformation comes from the standard \(J \Sigma J^T\) -esque covar propagation \[\sigma_{d_1}^2 = (\frac{d_1}{d_0}^4 \sigma^2_{d_0} + \sigma^2_p)\]
- \(\sigma^2_p\) is the propagation error (as in Kalman Filters )
- pixel coords are rounded to integer pixel indices
-
If there are more than 1 depth hypothesis for a pixel
- incorporate them if they're within \(2\sigma\) from each other
- ignore the farther observation otherwise
Depth map regularization
Local smoothing
- weighted average depths among all pixel neighbors that are within \(2\sigma\)
Outlier removal
-
for each invdepth hypothesis, track a validity score
- probability that the hypothesis is invalid or an outlier
- decrease this probability for every successful stereo observation
- increase for every failed match
- if p(all neighbors invalid) > threshold (in the joint prob sense), remove the hypothesis
- if p(all neighborsinvalid ) < threshold, add new hypothesis using variance weighted average of the neighbors
Dense Tracking
\[E(e) = \sum_i \frac{\alpha(r_i(e))}{\sigma_{d_i}^2}r_i\]
- alpha is the weight for a residual, \(\alpha \colon \mathbb{R} \rightarrow \mathbb{R}\)
- \(e \in \text{sim3}\)
- \(r_i(e)\) is the residual for the \(i^{\text{th}}\) pixel \[r_i(e) = (I_2(w(x_i, d_i, e)) - I_1(x_i))^2\]
- \(w\) warps the 3d point corresponding to \(x_i, d_i\) into image \(I_2\)
- Solve using iteratively weighted Gauss-Newton, 4 pyramid levels, coarse to fine
- Details in the 20 years of Lucas-Kanade paper
System Overview
- Tracking thread tracks new frames based on the last KeyFrame's depth map at 30Hz
- Depth estimation thread propagates inverse depths to last KeyFrame at 15Hz