  Multilingual BERT (mBERT) has demonstrated considerable cross-lingual
syntactic ability, whereby it enables effective zero-shot cross-lingual
transfer of syntactic knowledge. The transfer is more successful between some
languages, but it is not well understood what leads to this variation and
whether it fairly reflects difference between languages. In this work, we
investigate the distributions of grammatical relations induced from mBERT in
the context of 24 typologically different languages. We demonstrate that the
distance between the distributions of different languages is highly consistent
with the syntactic difference in terms of linguistic formalisms. Such
difference learnt via self-supervision plays a crucial role in the zero-shot
transfer performance and can be predicted by variation in morphosyntactic
properties between languages. These results suggest that mBERT properly encodes
languages in a way consistent with linguistic diversity and provide insights
into the mechanism of cross-lingual transfer.
