  Variants of the BERT architecture specialised for producing full-sentence
representations often achieve better performance on downstream tasks than
sentence embeddings extracted from vanilla BERT. However, there is still little
understanding of what properties of inputs determine the properties of such
representations. In this study, we construct several sets of sentences with
pre-defined lexical and syntactic structures and show that SOTA sentence
transformers have a strong nominal-participant-set bias: cosine similarities
between pairs of sentences are more strongly determined by the overlap in the
set of their noun participants than by having the same predicates, lengthy
nominal modifiers, or adjuncts. At the same time, the precise
syntactic-thematic functions of the participants are largely irrelevant.
