  Learning-based visual relocalizers exhibit leading pose accuracy, but require
hours or days of training. Since training needs to happen on each new scene
again, long training times make learning-based relocalization impractical for
most applications, despite its promise of high accuracy. In this paper we show
how such a system can actually achieve the same accuracy in less than 5
minutes. We start from the obvious: a relocalization network can be split in a
scene-agnostic feature backbone, and a scene-specific prediction head. Less
obvious: using an MLP prediction head allows us to optimize across thousands of
view points simultaneously in each single training iteration. This leads to
stable and extremely fast convergence. Furthermore, we substitute effective but
slow end-to-end training using a robust pose solver with a curriculum over a
reprojection loss. Our approach does not require privileged knowledge, such a
depth maps or a 3D model, for speedy training. Overall, our approach is up to
300x faster in mapping than state-of-the-art scene coordinate regression, while
keeping accuracy on par.
