  In this paper we present CMRNet, a realtime approach based on a Convolutional
Neural Network to localize an RGB image of a scene in a map built from LiDAR
data. Our network is not trained in the working area, i.e. CMRNet does not
learn the map. Instead it learns to match an image to the map. We validate our
approach on the KITTI dataset, processing each frame independently without any
tracking procedure. CMRNet achieves 0.27m and 1.07deg median localization
accuracy on the sequence 00 of the odometry dataset, starting from a rough pose
estimate displaced up to 3.5m and 17deg. To the best of our knowledge this is
the first CNN-based approach that learns to match images from a monocular
camera to a given, preexisting 3D LiDAR-map.
