  We focus on the problem of search in the multilingual setting. Examining the
problems of next-sentence prediction and inverse cloze, we show that at large
scale, instance-based transfer learning is surprisingly effective in the
multilingual setting, leading to positive transfer on all of the 35 target
languages and two tasks tested. We analyze this improvement and argue that the
most natural explanation, namely direct vocabulary overlap between languages,
only partially explains the performance gains: in fact, we demonstrate
target-language improvement can occur after adding data from an auxiliary
language even with no vocabulary in common with the target. This surprising
result is due to the effect of transitive vocabulary overlaps between pairs of
auxiliary and target languages.
