  The impressive achievements of transformers force NLP researchers to delve
into how these models represent the underlying structure of natural language.
In this paper, we propose a novel standpoint to investigate the above issue:
using typological similarities among languages to observe how their respective
monolingual models encode structural information. We aim to layer-wise compare
transformers for typologically similar languages to observe whether these
similarities emerge for particular layers. For this investigation, we propose
to use Centered Kernel Alignment to measure similarity among weight matrices.
We found that syntactic typological similarity is consistent with the
similarity between the weights in the middle layers, which are the pretrained
BERT layers to which syntax encoding is generally attributed. Moreover, we
observe that a domain adaptation on semantically equivalent texts enhances this
similarity among weight matrices.
