  Multilingual sentence encoders have seen much success in cross-lingual model
transfer for downstream NLP tasks. Yet, we know relatively little about the
properties of individual languages or the general patterns of linguistic
variation that they encode. We propose methods for probing sentence
representations from state-of-the-art multilingual encoders (LASER, M-BERT, XLM
and XLM-R) with respect to a range of typological properties pertaining to
lexical, morphological and syntactic structure. In addition, we investigate how
this information is distributed across all layers of the models. Our results
show interesting differences in encoding linguistic variation associated with
different pretraining strategies.
