  This paper explores the topic of transportability, as a sub-area of
generalisability. By proposing the utilisation of metrics based on
well-established statistics, we are able to estimate the change in performance
of NLP models in new contexts. Defining a new measure for transportability may
allow for better estimation of NLP system performance in new domains, and is
crucial when assessing the performance of NLP systems in new tasks and domains.
Through several instances of increasing complexity, we demonstrate how
lightweight domain similarity measures can be used as estimators for the
transportability in NLP applications. The proposed transportability measures
are evaluated in the context of Named Entity Recognition and Natural Language
Inference tasks.
