  Compared to the onboard camera and laser scanner, radar sensor provides
lighting and weather invariant sensing, which is naturally suitable for
long-term localization under adverse conditions. However, radar data is sparse
and noisy, resulting in challenges for radar mapping. On the other hand, the
most popular available map currently is built by lidar. In this paper, we
propose an end-to-end deep learning framework for Radar Localization on Lidar
Map (RaLL) to bridge the gap, which not only achieves the robust radar
localization but also exploits the mature lidar mapping technique, thus
reducing the cost of radar mapping. We first embed both sensor modals into a
common feature space by a neural network. Then multiple offsets are added to
the map modal for exhaustive similarity evaluation against the current radar
modal, yielding the regression of the current pose. Finally, we apply this
differentiable measurement model to a Kalman Filter (KF) to learn the whole
sequential localization process in an end-to-end manner. \textit{The whole
learning system is differentiable with the network based measurement model at
the front-end and KF at the back-end.} To validate the feasibility and
effectiveness, we employ multi-session multi-scene datasets collected from the
real world, and the results demonstrate that our proposed system achieves
superior performance over $90km$ driving, even in generalization scenarios
where the model training is in UK, while testing in South Korea. We also
release the source code publicly.
