  Word vectors and Language Models (LMs) pretrained on a large amount of
unlabelled data can dramatically improve various Natural Language Processing
(NLP) tasks. However, the measure and impact of similarity between pretraining
data and target task data are left to intuition. We propose three
cost-effective measures to quantify different aspects of similarity between
source pretraining and target task data. We demonstrate that these measures are
good predictors of the usefulness of pretrained models for Named Entity
Recognition (NER) over 30 data pairs. Results also suggest that pretrained LMs
are more effective and more predictable than pretrained word vectors, but
pretrained word vectors are better when pretraining data is dissimilar.
