Paper: https://arxiv.org/abs/1911.11763
Idea
- Finding correct correspondences between two images is a fundamental and challenging step in SLAM and SfM pipelines.
- SuperGlue is a middle-end that learns to match features.
-
Existing feature matching approaches have a few issues.
-
classical approach - detect keypoints, compute descriptors, nearest
neighbor match, filter matches, use RANSAC to estimate F/E and filter
outliers.
- Location of keypoints is ignored
- Filtering with something like Lowe's ratio test is crude
- Some deep learning methods use NN matches and filter using outlier prediction, thus starting with the same NN pitfalls
- Other works that learn matching match dense points or 3D point clouds, and also suffer from the same pitfalls
-
classical approach - detect keypoints, compute descriptors, nearest
neighbor match, filter matches, use RANSAC to estimate F/E and filter
outliers.
- There are ideas from deep learning on graphs and sets that the authors bring into SuperGlue
- Humans look back and forth between images to match correspondences, comparing between choices and refining matches, which suggests that an iterative approach with context aggregation should help with better matching
- Local features are extracted from two images A and B. Local feature is a joint vector formed by the keypoint location p and descriptor d of a keypoint.
- The goal is to compute partial assignment that match the M local features in A to the N features in B
- A matrix P(MXN) represents this assignment, with each element P MN being the confidence that M and N correspond to the same real world point
- SuperGlue has two layers - Attentional GNN and an Optimal Matching Layer
Attentional GNN
-
A multiplex graph is built, with two two types of edges
- self edges connect local feature i to all the other local features in the same image
- cross edges connect local feature i to all the local features in the other image
-
At each layer, the IR for keypoint x
i
is refined using
aggregation
-
x\
i
= x
i
+ MLP([xi | m])
- Note - Residual connection probably helps with preserving the input features at the later layers, for eg, later layers still have access to keypoint position
- m is the message produced by attention
- xi is the input representation for layer l
- x'\ i is the output representation
-
x\
i
= x
i
+ MLP([xi | m])
-
messages m are produced by attention
- for x i , a query is produced through linear projection
- for the rest of the keypoints, key and value are produced through linear projection
- query and keys are dot producted, products are then softmaxed, which give weights to each keypoints
- message is then sum of (softmax weight * value)
- multihead attention is used in practice
- final matching descriptors (f) are linear projections of the last layer's output
Optimal Matching Layer
- Takes the keypoint embeddings from GNN and produces a partial assignment matrix
- The pairwise score Si,j is the inner product of f A and f B
- To let the network suppress some keypoints, both sets of keypoints are augmented with an extra dummy keypoint, called dustbin, so that keypoints visible in one image and occluded in the other can be assigned to the dustbin
- Sinkhorn algorithm iterations produce the matrix P
Training
- Loss includes terms for maximizing the likelihood of correct matches, and correct occlusion prediction (making sure occluded points end up at the dustbin)
Results
- Much better pose estimation accuracy and matching scores than other methods
- Using superpoint instead of SIFT boosts metrics further
-
From Ablation
- Graph neural network, cross attention, positional encoding all seem to be very useful
- GNN is especially effective even with only 3 layers
Ideas
- Instead of computing GNN passes for the whole multiplex graph, we can maybe filter out edges that are surely not true matches (very high descriptor differences) so that fewer edges pass into the pipeline