  Language models are ubiquitous in current NLP, and their multilingual
capacity has recently attracted considerable attention. However, current
analyses have almost exclusively focused on (multilingual variants of) standard
benchmarks, and have relied on clean pre-training and task-specific corpora as
multilingual signals. In this paper, we introduce XLM-T, a model to train and
evaluate multilingual language models in Twitter. In this paper we provide: (1)
a new strong multilingual baseline consisting of an XLM-R (Conneau et al. 2020)
model pre-trained on millions of tweets in over thirty languages, alongside
starter code to subsequently fine-tune on a target task; and (2) a set of
unified sentiment analysis Twitter datasets in eight different languages and a
XLM-T model fine-tuned on them.
