  Deep transformer models have pushed performance on NLP tasks to new limits,
suggesting sophisticated treatment of complex linguistic inputs, such as
phrases. However, we have limited understanding of how these models handle
representation of phrases, and whether this reflects sophisticated composition
of phrase meaning like that done by humans. In this paper, we present
systematic analysis of phrasal representations in state-of-the-art pre-trained
transformers. We use tests leveraging human judgments of phrase similarity and
meaning shift, and compare results before and after control of word overlap, to
tease apart lexical effects versus composition effects. We find that phrase
representation in these models relies heavily on word content, with little
evidence of nuanced composition. We also identify variations in phrase
representation quality across models, layers, and representation types, and
make corresponding recommendations for usage of representations from these
models.
