  Correspondence pruning aims to search consistent correspondences (inliers)
from a set of putative correspondences. It is challenging because of the
disorganized spatial distribution of numerous outliers, especially when
putative correspondences are largely dominated by outliers. It's more
challenging to ensure effectiveness while maintaining efficiency. In this
paper, we propose an effective and efficient method for correspondence pruning.
Inspired by the success of attentive context in correspondence problems, we
first extend the attentive context to the first-order attentive context and
then introduce the idea of attention in attention (ANA) to model second-order
attentive context for correspondence pruning. Compared with first-order
attention that focuses on feature-consistent context, second-order attention
dedicates to attention weights itself and provides an additional source to
encode consistent context from the attention map. For efficiency, we derive two
approximate formulations for the naive implementation of second-order attention
to optimize the cubic complexity to linear complexity, such that second-order
attention can be used with negligible computational overheads. We further
implement our formulations in a second-order context layer and then incorporate
the layer in an ANA block. Extensive experiments demonstrate that our method is
effective and efficient in pruning outliers, especially in high-outlier-ratio
cases. Compared with the state-of-the-art correspondence pruning approach
LMCNet, our method runs 14 times faster while maintaining a competitive
accuracy.
