  The success of multilingual pre-trained models is underpinned by their
ability to learn representations shared by multiple languages even in absence
of any explicit supervision. However, it remains unclear how these models learn
to generalise across languages. In this work, we conjecture that multilingual
pre-trained models can derive language-universal abstractions about grammar. In
particular, we investigate whether morphosyntactic information is encoded in
the same subset of neurons in different languages. We conduct the first
large-scale empirical study over 43 languages and 14 morphosyntactic categories
with a state-of-the-art neuron-level probe. Our findings show that the
cross-lingual overlap between neurons is significant, but its extent may vary
across categories and depends on language proximity and pre-training data size.
