  Multilingual pre-trained language models transfer remarkably well on
cross-lingual downstream tasks. However, the extent to which they learn
language-neutral representations (i.e., shared representations that encode
similar phenomena across languages), and the effect of such representations on
cross-lingual transfer performance, remain open questions. In this work, we
conceptualize language neutrality of multilingual models as a function of the
overlap between language-encoding sub-networks of these models. We employ the
lottery ticket hypothesis to discover sub-networks that are individually
optimized for various languages and tasks. Our evaluation across three distinct
tasks and eleven typologically-diverse languages demonstrates that sub-networks
for different languages are topologically similar (i.e., language-neutral),
making them effective initializations for cross-lingual transfer with limited
performance degradation.
