  Particle tracking is crucial for almost all physics analysis programs at the
Large Hadron Collider. Deep learning models are pervasively used in particle
tracking related tasks. However, the current practice is to design and train
one deep learning model for one task with supervised learning techniques. The
trained models work well for tasks they are trained on but show no or little
generalization capabilities. We propose to unify these models with a language
model. In this paper, we present a tokenized detector representation that
allows us to train a BERT model for particle tracking. The trained BERT model,
namely TrackingBERT, offers latent detector module embedding that can be used
for other tasks. This work represents the first step towards developing a
foundational model for particle detector understanding.
