  This paper presents EstBERT, a large pretrained transformer-based
language-specific BERT model for Estonian. Recent work has evaluated
multilingual BERT models on Estonian tasks and found them to outperform the
baselines. Still, based on existing studies on other languages, a
language-specific BERT model is expected to improve over the multilingual ones.
We first describe the EstBERT pretraining process and then present the results
of the models based on finetuned EstBERT for multiple NLP tasks, including POS
and morphological tagging, named entity recognition and text classification.
The evaluation results show that the models based on EstBERT outperform
multilingual BERT models on five tasks out of six, providing further evidence
towards a view that training language-specific BERT models are still useful,
even when multilingual models are available.
