  Cross-lingual document classification aims at training a document classifier
on resources in one language and transferring it to a different language
without any additional resources. Several approaches have been proposed in the
literature and the current best practice is to evaluate them on a subset of the
Reuters Corpus Volume 2. However, this subset covers only few languages
(English, German, French and Spanish) and almost all published works focus on
the the transfer between English and German. In addition, we have observed that
the class prior distributions differ significantly between the languages. We
argue that this complicates the evaluation of the multilinguality. In this
paper, we propose a new subset of the Reuters corpus with balanced class priors
for eight languages. By adding Italian, Russian, Japanese and Chinese, we cover
languages which are very different with respect to syntax, morphology, etc. We
provide strong baselines for all language transfer directions using
multilingual word and sentence embeddings respectively. Our goal is to offer a
freely available framework to evaluate cross-lingual document classification,
and we hope to foster by these means, research in this important area.
