  A recent introduction of Transformer deep learning architecture made
breakthroughs in various natural language processing tasks. However,
non-English languages could not leverage such new opportunities with the
English text pre-trained models. This changed with research focusing on
multilingual models, where less-spoken languages are the main beneficiaries. We
compare pre-trained multilingual BERT, XLM-R, and older learned text
representation methods as encodings for the task of Lithuanian news clustering.
Our results indicate that publicly available pre-trained multilingual
Transformer models can be fine-tuned to surpass word vectors but still score
much lower than specially trained doc2vec embeddings.
