  This paper presents a novel methodology for improving the performance of
machine learning based space traffic management tasks through the use of a
pre-trained orbit model. Taking inspiration from BERT-like self-supervised
language models in the field of natural language processing, we introduce
ORBERT, and demonstrate the ability of such a model to leverage large
quantities of readily available orbit data to learn meaningful representations
that can be used to aid in downstream tasks. As a proof of concept of this
approach we consider the task of all vs. all conjunction screening, phrased
here as a machine learning time series classification task. We show that
leveraging unlabelled orbit data leads to improved performance, and that the
proposed approach can be particularly beneficial for tasks where the
availability of labelled data is limited.
