  The task of online mapping is to predict a local map using current sensor
observations, e.g. from lidar and camera, without relying on a pre-built map.
State-of-the-art methods are based on supervised learning and are trained
predominantly using two datasets: nuScenes and Argoverse 2. However, these
datasets revisit the same geographic locations across training, validation, and
test sets. Specifically, over $80$% of nuScenes and $40$% of Argoverse 2
validation and test samples are less than $5$ m from a training sample. At test
time, the methods are thus evaluated more on how well they localize within a
memorized implicit map built from the training data than on extrapolating to
unseen locations. Naturally, this data leakage causes inflated performance
numbers and we propose geographically disjoint data splits to reveal the true
performance in unseen environments. Experimental results show that methods
perform considerably worse, some dropping more than $45$ mAP, when trained and
evaluated on proper data splits. Additionally, a reassessment of prior design
choices reveals diverging conclusions from those based on the original split.
Notably, the impact of lifting methods and the support from auxiliary tasks
(e.g., depth supervision) on performance appears less substantial or follows a
different trajectory than previously perceived. Splits can be found at
https://github.com/LiljaAdam/geographical-splits
