  Many studies were recently done for investigating the properties of
contextual language models but surprisingly, only a few of them consider the
properties of these models in terms of semantic similarity. In this article, we
first focus on these properties for English by exploiting the distributional
principle of substitution as a probing mechanism in the controlled context of
SemCor and WordNet paradigmatic relations. Then, we propose to adapt the same
method to a more open setting for characterizing the differences between static
and contextual language models.
