  Contextualized word embeddings have demonstrated state-of-the-art performance
in various natural language processing tasks including those that concern
historical semantic change. However, language models such as BERT was trained
primarily on contemporary corpus data. To investigate whether training on
historical corpus data improves diachronic semantic analysis, we present a
pre-trained BERT-based language model, HistBERT, trained on the balanced Corpus
of Historical American English. We examine the effectiveness of our approach by
comparing the performance of the original BERT and that of HistBERT, and we
report promising results in word similarity and semantic shift analysis. Our
work suggests that the effectiveness of contextual embeddings in diachronic
semantic analysis is dependent on the temporal profile of the input text and
care should be taken in applying this methodology to study historical semantic
change.
