  Word embeddings and pre-trained language models allow to build rich
representations of text and have enabled improvements across most NLP tasks.
Unfortunately they are very expensive to train, and many small companies and
research groups tend to use models that have been pre-trained and made
available by third parties, rather than building their own. This is suboptimal
as, for many languages, the models have been trained on smaller (or lower
quality) corpora. In addition, monolingual pre-trained models for non-English
languages are not always available. At best, models for those languages are
included in multilingual versions, where each language shares the quota of
substrings and parameters with the rest of the languages. This is particularly
true for smaller languages such as Basque. In this paper we show that a number
of monolingual models (FastText word embeddings, FLAIR and BERT language
models) trained with larger Basque corpora produce much better results than
publicly available versions in downstream NLP tasks, including topic
classification, sentiment classification, PoS tagging and NER. This work sets a
new state-of-the-art in those tasks for Basque. All benchmarks and models used
in this work are publicly available.
