  There is an increasing amount of evidence that in cases with little or no
data in a target language, training on a different language can yield
surprisingly good results. However, currently there are no established
guidelines for choosing the training (source) language. In attempt to solve
this issue we thoroughly analyze a state-of-the-art multilingual model and try
to determine what impacts good transfer between languages. As opposed to the
majority of multilingual NLP literature, we don't only train on English, but on
a group of almost 30 languages. We show that looking at particular syntactic
features is 2-4 times more helpful in predicting the performance than an
aggregated syntactic similarity. We find out that the importance of syntactic
features strongly differs depending on the downstream task - no single feature
is a good performance predictor for all NLP tasks. As a result, one should not
expect that for a target language $L_1$ there is a single language $L_2$ that
is the best choice for any NLP task (for instance, for Bulgarian, the best
source language is French on POS tagging, Russian on NER and Thai on NLI). We
discuss the most important linguistic features affecting the transfer quality
using statistical and machine learning methods.
