  Image-to-point cloud registration aims to determine the relative camera pose
between an RGB image and a reference point cloud, serving as a general solution
for locating 3D objects from 2D observations. Matching individual points with
pixels can be inherently ambiguous due to modality gaps. To address this
challenge, we propose a framework to capture quantity-aware correspondences
between local point sets and pixel patches and refine the results at both the
point and pixel levels. This framework aligns the high-level semantics of point
sets and pixel patches to improve the matching accuracy. On a coarse scale, the
set-to-patch correspondence is expected to be influenced by the quantity of 3D
points. To achieve this, a novel supervision strategy is proposed to adaptively
quantify the degrees of correlation as continuous values. On a finer scale,
point-to-pixel correspondences are refined from a smaller search space through
a well-designed scheme, which incorporates both resampling and quantity-aware
priors. Particularly, a confidence sorting strategy is proposed to
proportionally select better correspondences at the final stage. Leveraging the
advantages of high-quality correspondences, the problem is successfully
resolved using an efficient Perspective-n-Point solver within the framework of
random sample consensus (RANSAC). Extensive experiments on the KITTI Odometry
and NuScenes datasets demonstrate the superiority of our method over the
state-of-the-art methods.
