  We empirically demonstrate that a transformer pre-trained on country-scale
unlabeled human mobility data learns embeddings capable, through fine-tuning,
of developing a deep understanding of the target geography and its
corresponding mobility patterns. Utilizing an adaptation framework, we evaluate
the performance of our pre-trained embeddings in encapsulating a broad spectrum
of concepts directly and indirectly related to human mobility. This includes
basic notions, such as geographic location and distance, and extends to more
complex constructs, such as administrative divisions and land cover. Our
extensive empirical analysis reveals a substantial performance boost gained
from pre-training, reaching up to 38% in tasks such as tree-cover regression.
We attribute this result to the ability of the pre-training to uncover
meaningful patterns hidden in the raw data, beneficial for modeling relevant
high-level concepts. The pre-trained embeddings emerge as robust
representations of regions and trajectories, potentially valuable for a wide
range of downstream applications.
