  Multilingual language models such as mBERT have seen impressive cross-lingual
transfer to a variety of languages, but many languages remain excluded from
these models. In this paper, we analyse the effect of pre-training with
monolingual data for a low-resource language that is not included in mBERT --
Maltese -- with a range of pre-training set ups. We conduct evaluations with
the newly pre-trained models on three morphosyntactic tasks -- dependency
parsing, part-of-speech tagging, and named-entity recognition -- and one
semantic classification task -- sentiment analysis. We also present a newly
created corpus for Maltese, and determine the effect that the pre-training data
size and domain have on the downstream performance. Our results show that using
a mixture of pre-training domains is often superior to using Wikipedia text
only. We also find that a fraction of this corpus is enough to make significant
leaps in performance over Wikipedia-trained models. We pre-train and compare
two models on the new corpus: a monolingual BERT model trained from scratch
(BERTu), and a further pre-trained multilingual BERT (mBERTu). The models
achieve state-of-the-art performance on these tasks, despite the new corpus
being considerably smaller than typically used corpora for high-resourced
languages. On average, BERTu outperforms or performs competitively with mBERTu,
and the largest gains are observed for higher-level tasks.
