  Image-to-point cloud registration aims to determine the relative camera pose
of an RGB image with respect to a point cloud. It plays an important role in
camera localization within pre-built LiDAR maps. Despite the modality gaps,
most learning-based methods establish 2D-3D point correspondences in feature
space without any feedback mechanism for iterative optimization, resulting in
poor accuracy and interpretability. In this paper, we propose to reformulate
the registration procedure as an iterative Markov decision process, allowing
for incremental adjustments to the camera pose based on each intermediate
state. To achieve this, we employ reinforcement learning to develop a
cross-modal registration agent (CMR-Agent), and use imitation learning to
initialize its registration policy for stability and quick-start of the
training. According to the cross-modal observations, we propose a 2D-3D hybrid
state representation that fully exploits the fine-grained features of RGB
images while reducing the useless neutral states caused by the spatial
truncation of camera frustum. Additionally, the overall framework is
well-designed to efficiently reuse one-shot cross-modal embeddings, avoiding
repetitive and time-consuming feature extraction. Extensive experiments on the
KITTI-Odometry and NuScenes datasets demonstrate that CMR-Agent achieves
competitive accuracy and efficiency in registration. Once the one-shot
embeddings are completed, each iteration only takes a few milliseconds.
