  Language models based on deep neural networks have facilitated great advances
in natural language processing and understanding tasks in recent years. While
models covering a large number of languages have been introduced, their
multilinguality has come at a cost in terms of monolingual performance, and the
best-performing models at most tasks not involving cross-lingual transfer
remain monolingual. In this paper, we consider the question of whether it is
possible to pre-train a bilingual model for two remotely related languages
without compromising performance at either language. We collect pre-training
data, create a Finnish-English bilingual BERT model and evaluate its
performance on datasets used to evaluate the corresponding monolingual models.
Our bilingual model performs on par with Google's original English BERT on GLUE
and nearly matches the performance of monolingual Finnish BERT on a range of
Finnish NLP tasks, clearly outperforming multilingual BERT. We find that when
the model vocabulary size is increased, the BERT-Base architecture has
sufficient capacity to learn two remotely related languages to a level where it
achieves comparable performance with monolingual models, demonstrating the
feasibility of training fully bilingual deep language models. The model and all
tools involved in its creation are freely available at
https://github.com/TurkuNLP/biBERT
