  The commonly adopted detect-then-match approach to registration finds
difficulties in the cross-modality cases due to the incompatible keypoint
detection and inconsistent feature description. We propose, 2D3D-MATR, a
detection-free method for accurate and robust registration between images and
point clouds. Our method adopts a coarse-to-fine pipeline where it first
computes coarse correspondences between downsampled patches of the input image
and the point cloud and then extends them to form dense correspondences between
pixels and points within the patch region. The coarse-level patch matching is
based on transformer which jointly learns global contextual constraints with
self-attention and cross-modality correlations with cross-attention. To resolve
the scale ambiguity in patch matching, we construct a multi-scale pyramid for
each image patch and learn to find for each point patch the best matching image
patch at a proper resolution level. Extensive experiments on two public
benchmarks demonstrate that 2D3D-MATR outperforms the previous state-of-the-art
P2-Net by around $20$ percentage points on inlier ratio and over $10$ points on
registration recall. Our code and models are available at
https://github.com/minhaolee/2D3DMATR.
